ISSUES&ANSWERS 



REL 2007-No. 008 




Analysis 
of Title IIB 
Mathematics 
and Science 
Partnerships in 
the Northwest 
Region 





i 







NATIONAL CENTER for 
EDUCATION EVALUATION 
AND REGIONAL ASSISTANCE 



I 



Institute of Education Sciences 
U.S. Department of Education 







REL 2007-No. 008 



ISSUES^ANSWERS 



:*REL 

NORTHWEST 

Regional Educational Laboratory 
At Northwest Regional 
Educational Laboratory 

Analysis of Title IIB Mathematics 
and Science Partnerships in 
the Northwest Region 



June 2007 



Prepared by 
Edith Gummer 

Northwest Regional Educational Laboratory 
Jennifer Stepanek 

Northwest Regional Educational Laboratory 







NATIONAL CENTER for 
EDUCATION EVALUATION 
AND REGIONAL ASSISTANCE 



Institute of Education Sciences 
U.S. Department of Education 




NORTHWEST 




Issues & Answers is an ongoing series of reports from short-term Fast Response Projects conducted by the regional educa- 
tional laboratories on current education issues of importance at local, state, and regional levels. Fast Response Project topics 
change to reflect new issues, as identified through lab outreach and requests for assistance from policymakers and educa- 
tors at state and local levels and from communities, businesses, parents, families, and youth. All Issues & Answers reports 
meet Institute of Education Sciences standards for scientifically valid research. 

June 2007 

This report was prepared for the Institute of Education Sciences (lES) under Contract ED-06-CO-002 by Regional Educa- 
tional Laboratory Northwest administered by Northwest Regional Educational Laboratory. The content of the publication 
does not necessarily reflect the views or policies of lES or the U.S. Department of Education nor does mention of trade 
names, commercial products, or organizations imply endorsement by the U.S. Government. 

This report is in the public domain. While permission to reprint this publication is not necessary, it should be cited as: 

Gummer, E., & Stepanek, J. (2007). Analysis of Title IIB Mathematics and Science Partnerships in the Northwest Region 
(Issues & Answers Report, REL 2007-No. 008). Washington, DC: U.S. Department of Education, Institute of Education 
Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northwest. 
Retrieved from http://ies.ed.gov/ncee/edlabs. 

This report is available on the regional educational laboratory web site at http://ies.ed.gov/ncee/edlabs. 



Ml 



Summary 

Analysis of Title IIB Mathematics 
and Science Partnerships in 
the Northwest Region 



This report describes the first year of the 
funded professional development activi- 
ties in the Title IIB Math and Science Part- 
nership projects in the Northwest Region 
and the evaluation models. The analysis 
is structured around the factors of profes- 
sional development associated with chang- 
es in teacher knowledge and practice. 

Title IIB Mathematics and Science Partner- 
ships (MSPs) are the main resource in the No 
Child Left Behind Act to support the ongoing 
professional development of science and math- 
ematics teachers. Funds available to states must 
be used to purchase high-quality professional 
development. In addition, with increasing 
concerns about accountability throughout the 
field — from federal agencies to the individual 
classroom teacher and student — educational 
interventions must demonstrate a positive im- 
pact on important educational outcomes. The 
Title IIB MSPs are intended to positively affect 
content knowledge and pedagogical skills for 
mathematics and science teachers. The ulti- 
mate goal is improved student achievement in 
mathematics and science. 

This report describes the nature of the funded 
professional development activities in the Title 
IIB MSP projects in the Northwest Region and 
characterizes the models of evaluation during 
their first year of implementation, 2004-05. 



The analysis is structured around the factors 
of professional development that have been 
identified as associated with changes in teacher 
knowledge and practice (Desimone et al., 2002; 
Caret, Birman et al, 1999; Caret, Porter et al., 
2001; Porter et al., 2000). The description of 
the evaluations examines the extent to which 
the projects have connected their activities to 
measurable outcomes for teacher knowledge 
and practice and for student achievement, mea- 
sured those outcomes, and clearly articulated 
their qualitative and quantitative study designs. 



All projects met at least some of the criteria for 
high-quality professional development 

The prevalent model of professional develop- 
ment in the MSP projects was a two-week, 
content-focused workshop or institute held 
during the summer, with follow-up support 
for teachers during the school year. One reason 
that this model was so common is that three of 
the five Northwest Region states required it in 
their requests for proposals. However, most of 
the projects in Idaho and Montana — the two 
states that did not require an institute — also 
conform to this model. This may be because 
the model is highlighted and defined in both 
the legislation and the requests for proposals, 
or it may also reflect the prevalence of the in- 
stitute model in the previously funded Eisen- 
hower Professional Development Program. 
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Although the summer institute was preva- 
lent, the projects in the Northwest Region did 
feature many variations on that model. Proj- 
ects differed in the way they structured and 
conducted the follow-up activities, as well as in 
the amount of ongoing support. Some projects 
included less emphasis on the summer work- 
shop and more on ongoing, school-embedded, 
and collaborative activities for teachers. De- 
spite these variations, the multiple authorized 
activities suggested in the legislation — such as 
recruitment of mathematics, engineering, and 
science majors to teaching through a variety 
of mechanisms— are not the focus of projects 
funded in the Northwest Region. 

All of the projects provided evidence in the 
documentation that they met at least some of 
the criteria for high-quality professional devel- 
opment. It is far from clear whether projects 
must meet all criteria in every category to be 
considered effective. No available evidence in- 
dicates that professional development projects 
are more effective when they are rated highly 
in all categories than when they receive high 
ratings in only some categories. Nor is there 
evidence that any criterion is more important 
or less important than the others. 



Evaluation presented significant challenges 
to the Title IIB MSP projects 

Evaluation design and implementation in year 
one of the Title IIB MSP projects were prob- 
lematic. In interviews many project staff and 
evaluators reported difficulties designing and 
implementing adequate evaluation due to the 
late awarding of the Title IIB MSP contracts 
for professional development. In addition, 
limited budgetary resources were identified as 
barriers to effective evaluation. 



Evaluations of many projects relied on capturing 
participant reactions and self-reporting as the 
only sources of evidence of their effectiveness. 
Few projects used well developed instruments 
to measure changes in teacher content knowl- 
edge. Projects indicated difficulties using state 
assessments to directly measure the impact of 
projects on student achievement. For instance, 
the professional development might include a 
majority of teachers who were teaching at a level 
different from that targeted by the state science 
assessment. The lack of instruments for measur- 
ing changes in teacher and student knowledge 
of specific content led some projects to attempt 
to develop their own measures, while other proj- 
ects resorted to less rigorous methods. 

Care should be taken in interpreting these 
findings because this analysis is based on the 
first year of implementing the Title IIB MSP 
programs, when evaluation designs may not be 
fully mature. However, the minimal extent to 
which the project evaluations addressed evalu- 
ation standards that should be well known in 
the evaluation and professional development 
community indicates larger issues than the 
barriers identified above. Clearly, there is room 
for improvement in the project evaluations. 

Ongoing technical assistance is necessary to 
increase the evaluation skills of the state edu- 
cation agency staff responsible for the Title IIB 
MSP programs and the staff and evaluators of 
the individual projects. The U.S. Department 
of Education regional forums are a start in 
informing stakeholders about the method and 
instruments to improve evaluations, but access 
to these regional forums is limited by project 
budget constraints. 

June 2007 



TABLE OF CONTENTS 



Summary iii 

Overview 1 

This report addresses three research questions 1 

Title IIB MSP programs vary across the five Northwest Region states 4 

The Title IIB program has multiple implications for analyzing how policy is translated into professional 
development program implementation 5 

What is the nature of the professional development provided by the Title IIB MSP projects in the Northwest 
Region? 5 

Duration provides opportunities for in-depth study and ongoing support 6 
Activity type has two dimensions: traditional and reform 6 

Collective participation includes activities for teachers from the same school or district 7 
Content focus addresses the substance of professional development 7 
Active learning describes participants’ learning experiences 8 

Coherence indicates how connected professional development is to other teacher learning and development 
activities 8 

There are four criteria for describing partnership 9 

Preconditions for partnership include existing relationships and mutual goals 9 

Complexity characterizes the structure of the partnership 9 

Interdependence describes how the partners organize their work 10 

Communication describes the frequency of contact among partners 10 

Only a brief comparison with the Council of Chief State School Officers project is possible 10 

What is the nature of the evaluations of the Title IIB MSP projects in the Northwest Region? 11 

Project context provides information about factors that might influence implementation 11 

Evaluation purpose should be clearly identified 1 1 

Evaluation questions should be aligned with project goals 12 

The identity and credibility of the evaluator should be clear 12 

Evaluation plans should describe stakeholder involvement 12 

Methodological approaches need to be clearly described 13 

Plans should report information sources and sampling 13 

Information about data collection instruments is essential 13 

Specific data collection procedures should be identified 14 

Descriptions of quantitative analysis processes should be detailed 14 

Qualitative analysis processes should be articulated 14 

Results from the evaluation analyses here cannot be compared with those from the Council of Chief State 
School Officers 15 

Under what conditions is the development of experimental or quasi-experimental models of evaluation 
appropriate and successful? 15 

Findings suggest future directions for research 16 

Notes 17 

Appendix A Analysis of the Title IIB Mathematics and Science Partnerships professional development 
projects by state 18 



vi 



Appendix B Analysis of the Title IIB MSP Professional Development evaluation projects by state 42 
Appendix C Interview protocol 48 
Appendix D Methodology 50 

Appendix E Comparison of professional development criteria: Council of Chief State School Officers and 
Northwest Regional Educational Laboratory 57 

References 58 

Boxes 

1 Details of the Title IIB Mathematics and Science Partnership legislation 2 

2 Details of the Improving Evaluation of Professional Development in Mathematics and Science Education 
project 3 

Figures 

1 Most projects provided 80 or more hours of professional development 6 

2 More than half of projects included at least some reform activities 7 

3 Almost two-thirds of projects included collaboration among teachers from the same school or district 7 

4 Most projects at least somewhat addressed content knowledge and teaching specific content 7 

5 Most projects provided active learning in the form of planning for implementation 8 

6 Ongoing communication was the most common dimension of coherence addressed by projects 9 

7 Most activities were designed to produce outcomes only for teachers and schools 9 

8 More projects had a vertical structure than had a horizontal structure 9 

9 More than half the projects served a large geographic region 10 

10 A pooled approach to collaboration was more common than a reciprocal approach 10 

11 Just over half of projects provided information about frequency of communication in the 

documentation 10 

12 Less than half of projects included information about the project context that might have influenced project 

implementation 11 

13 Few projects included details on how the projects used the information from the evaluation 12 

14 Few projects had evaluation questions that were clearly articulated, well aligned with project goals and 
objectives, and explicitly tied to data sources 12 

15 More than a third of projects identified a specific person for the evaluation and included that person’s 
qualifications 12 

16 Seven project evaluations mentioned stakeholder involvement 13 

17 Over half the projects included some information about the design of the evaluation 13 



18 More than half of projects identified the sources of information 13 

19 Most projects provided some information about instruments they used to collect data for their 
reports 14 

20 Few projects provided complete information on when and how data were collected 14 

21 Most evaluations did not provide complete information on quantitative analysis 14 

22 Only one project provided any information about how qualitative data were analyzed 15 

Tables 

A1 Journeys in Mathematics project features 19 

A2 Classroom Assessment for Mathematics Performance project features 21 

A3 Coeur d’Alene project features 22 

A4 Developing Mathematical Thinking project features 23 

A5 Twin Falls project features 24 

A6 Billings project features 25 

A7 Creating Opportunities in Mathematics for Exemplary Teaching project features 27 

A8 Flathead and Salish Kootenai College project features 28 

A9 High School Chemistry project features 29 

AlO Improving Science Teaching project features 30 

All Montana Science and Mathematics Consortium project features 31 

A12 Greater Oregon Mathematics Partnership project features 33 

A13 High Desert project features 34 

A14 Northwest Regional Education Service District project features 35 
A15 Willamette Valley Watershed Partnership Project features 36 
A16 Eastern Washington project features 38 

A17 Partnership for Reform in Secondary Science and Mathematics project features 39 
A18 Watershed Investigation Partnership project features 40 
A19 Whatcom and Skagit project features 41 

D1 Data sources 50 

D2 Analysis framework — professional development 51 

D3 Analysis framework — partnership 53 

D4 Analysis framework — evaluation 55 



OVERVIEW 



1 



This report 
describes the 
first year of 
the funded 
professional 
development 
activities in the 
Title IIB Math 
and Science 
Partnership 
projects in the 
Northwest Region 
and the evaluation 
models. The 
analysis is 
structured around 
the factors of 
professional 
development 
associated with 
changes in teacher 
knowledge 
and practice. 



OVERVIEW 

The Title IIB Mathematics and Science Partner- 
ship (MSP) program is a federally funded formula 
grant program to the states that represents a major 
investment in the professional development of 
mathematics and science teachers under the No 
Child Left Behind Act. The program supports 
partnerships between the mathematics, science, 
and engineering faculty of higher education 
institutions and high-need school districts to 
increase teachers’ content knowledge and peda- 
gogical skills. Other partners, including schools of 
education, business, and nonprofit organizations, 
may also join the work. In fiscal year (FY) 2003 an 
appropriation of just under $100 million supported 
more than 300 projects. In FY2004 $150 million 
was appropriated and awarded to the states in fall 
2004 and winter 2005. 

Each state was responsible for administering a 
competitive grants program with the federal funds 
(box 1). As part of this process, the states estab- 
lished program priorities and a review process to 
award projects that responded to the most press- 
ing problems in science and mathematics educa- 
tion that could be addressed through professional 
development. After funding the projects, the states 
were responsible for monitoring progress and 
working with the U.S. Department of Education to 
document their effectiveness. The enabling legisla- 
tion required states to annually report to the U.S. 
secretary of education on the programs’ impact 
on teachers’ content knowledge and on student 
learning. 



This report addresses three research questions 

This study is intended to provide policymak- 
ers, state agency staff, and university and school 
personnel interested in potential Title IIB MSP 
projects with information about how the MSP 
program has been implemented and evaluated 
in the Northwest Region in the first cohort of the 
funded projects. Results of the Northwest Re- 
gional Educational Laboratory needs assessment 
document the growing concern by educators in 
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BOX1 

Details of the Title IIB 
Mathematics and Science 
Partnership legislation 

The Title IIB MSP legislation pro- 
vides considerable guidance for how 
the Title IIB programs in the states 
should be structured to improve 
teacher quality in science and 
mathematics. 

Five goals for improving teaching 

The legislation has five goals for 
improving the teaching force in K-12 
science and mathematics education: 

• Improving and upgrading the 
status and stature of mathematics 
and science teaching by encour- 
aging higher education institu- 
tions to assume more responsibil- 
ity for improving mathematics 
and science teacher education 
through a comprehensive, 
integrated system of recruiting, 
training, and advising math- 
ematics and science teachers. 

• Focusing on career-long intellec- 
tual growth of teachers and up- 
grading of skills and knowledge. 

• Bringing mathematics and 
science teachers at both the 
elementary and secondary levels 
together with scientists, math- 
ematicians, and engineers to 
increase teachers’ subject matter 
knowledge. 

• Developing more rigor in science 
and mathematics curricula to 
align them with state and local 
academic content standards and 
with the standards expected for 
postsecondary study in engineer- 
ing, mathematics, and science. 



• Improving and expanding train- 
ing of mathematics and science 
teachers, including training 

in the effective integration of 
technology into curricula and 
instruction (Title IIB, Section 
2201, Purpose, Definitions). 

Required partners 

The legislation identifies the mem- 
bers of the partnership, including 
a state education agency; science, 
engineering, and mathemat- 
ics faculty at a higher education 
institutions; and a high-need local 
education agency. Other partners 
might include additional science, 
engineering, and mathematics 
faculty; teacher preparation faculty; 
additional local education agencies 
(public charter or private schools); 
businesses; and nonprofit or for- 
profit organizations. 

Authorized activities 

Although the legislation defines 
one possible model of professional 
development— the summer work- 
shop or institute— a range of 10 
authorized activities are suggested, 
including: 

• Professional development activi- 
ties that increase mathematics 
and science content knowledge 
for teachers. 

• Recruitment of mathematics, 
engineering, and science majors 
to teaching through a variety of 
mechanisms. 

• Development of rigorous science 
and mathematics curricula. 

• Development of distance learn- 
ing programs for mathematics 
and science teachers. 



• Design of programs to connect 
teachers to practicing scientists. 

• Development of teachers and pro- 
grams to encourage women and 
underrepresented populations in 
postsecondary study of science, 
technology, engineering and 
mathematics careers. (Title IIB, 
Section 2201, Grants for Mathe- 
matics and Science Partnerships). 

The definition of a summer workshop 
or institute includes direct interac- 
tions of at least two weeks between 
teachers, as students, and higher 
education institution faculty are re- 
quired. In addition, the workshop or 
institute must include at least three 
days of follow-up in the classroom or 
through distance learning. 

Evaluation and accountability plan 

MSP partnerships are required to 
develop an evaluation and account- 
ability plan that measures the impact 
of activities. The legislation clearly 
articulates that the evaluation needs to 
focus on the impact of the professional 
development on student achievement 
and must include measurable objec- 
tives to increase the number of teach- 
ers who participate in content-based 
professional development (Title IIB, 
Section 2201, Grants for Mathematics 
and Science Partnerships). 

Evaluations might also include 
measurable objectives to increase 
participation by students in advanced 
courses in mathematics and science 
and to increase percentages of teach- 
ers with academic majors or minors 
in science, technology, engineering, 
and mathematics or classes taught by 
such teachers. 
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the Northwest Region on implementing research- 
based instructional practices in core subject areas 
to directly improve student achievement. Respon- 
dents recognized that more effort is needed in 
identifying “research-based” best practices, and a 
majority said that they are particularly interested 
in professional development in specific research- 
based mathematics practices. The No Child Left 
Behind Act requires science assessments begin- 
ning in 2007-08, and 7 of 10 principals indicated 
that their schools need to put more effort into 
student proficiency in science. However, responses 
regarding professional development needs sug- 
gest that most schools have not yet given much 
thought to specific practices or models they might 
consider. 



3. Under what conditions is the development of 
experimental or quasi-experimental models of 
evaluation appropriate and successful? 

Descriptive analyses were conducted of the year 
one Title IIB MSP projects in Alaska, Idaho, 
Montana, Oregon, and Washington (see appen- 
dixes A and B). These analyses are intended to 
present the projects as they existed at the end 
of the first year of funding viewed through the 
lenses of criteria for high-quality professional 
development and program evaluation. A re- 
lated study is currently being conducted by the 
Council of Chief State School Officers in a project 
funded by the National Science Foundation 
(box 2). 



Three research questions structure this report: 

1. What is the nature of the professional devel- 
opment provided by the Title IIB MSP projects 
in the Northwest Region? 

2. What is the nature of the evaluation of the 
Title IIB MSP projects in the Northwest 
Region? 



The analysis is based on the available documents 
produced by the states and funded projects and 
an interview structured around core program- 
matic issues of the professional development and 
evaluation designs (appendix C). The documents 
included state requests for proposals, funded 
proposals, and year one annual reports. These are 
static documents, collected before the projects 
unfolded. 



BOX 2 

Details of the Improving 
Evaluation of Professional 
Development in Mathematics 
and Science Education project 

A related study is currently being 
conducted by the Council of Chief 
State School Officers (CCSSO) in 
a project funded by the National 
Science Foundation. The purpose 
project is to identify professional 
development programs that meet 
criteria established by research 
and to report on the effects of the 
identified programs on improving 
teaching and learning in mathemat- 
ics and science. The CCSSO study 
convened a review panel to examine 



27 nominated programs from 15 
states using a program quality rubric 
to assess the professional develop- 
ment and evaluation designs (Blank, 
2006). The preliminary results are 
available online at www.ccsso.org/ 
projects/Improving_Evaluation_of_ 
Professional_Development/. 

A detailed comparison of the results 
of the CCSSO review with the results 
of this study is not appropriate 
because the two projects are dif- 
ferent in both nature and scope. 

The CCSSO study is not limited to 
Title IIB MSPs and includes proj- 
ects funded by the National Sci- 
ence Foundation. Also, the CCSSO 
study conducted an analysis around 



iterative interviews with project staff 
and evaluators as well as project 
documents, while this study relies 
on a static database. Finally, the 
CCSSO study uses a rating system 
implemented by an expert panel, 
while this study is descriptive and 
uses similar but not identical criteria 
in its analysis. Although a detailed 
comparison is not possible, some 
general statements about the CCSSO 
findings are provided in this report. 
Overall, the preliminary findings 
from CCSSO suggest results similar 
to those of this analysis — namely, 
that there is variation among the 
projects in terms of both profes- 
sional development and evaluation 
design. 
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The nature of the professional development being 
implemented in each of the projects was exam- 
ined using an analytic framework based on the 
National Evaluation of the Eisenhower Profes- 
sional Development Program (Desimone et al., 
2002; Caret, Birman et al, 1999; Caret, Porter 
et al, 2001; Porter et al., 2000). The framework 
is organized around six features of high quality 
professional development that were identified 
in that evaluation of mathematics and science 
programs: duration, activity type, collective 
participation, content focus, active learning, 
and coherence (see appendix D). There are four 
dimensions of partnership: supporting precondi- 
tions, complexity, interdependence, and com- 
munication (Kingsley & O’Neil, 2004; Kingsley & 
Waschak, 2005). 

The analysis of the evaluations of Title IIB MSP 
projects used a matrix of evaluation criteria devel- 
oped by SRI International in the Online Evaluation 
Resource Library based on the program evalua- 
tion standards established by the Joint Committee 
on Standards for Educational Evaluation (JCSEE, 
1994). The program evaluation standards focus on 
the utility, feasibility, propriety, and accuracy of 
evaluations of educational programs. The Online 
Evaluation Resource Library web site includes 
three matrixes that provide a descriptive frame- 
work of what should be included in an evaluation 
that meets the program evaluation standards. 



Title IIB MSP programs vary across the 
five Northwest Region states 

Alaska is the only state among the five that funded 
only one project in the first year. The Alaska 
project served 51 teachers in two school districts: 
Anchorage— a large urban district— and Lake 
and Peninsula— a large rural district with many 
remote schools. The partnership for this project in- 
cluded the two districts and the University of Alas- 
ka-Anchorage. The Alaska MSP program focused 
on K-8 mathematics. Some of the requirements 
identified by the state included a summer institute 
of at least two weeks with follow-up during the 
school year, coursework to help teachers achieve 



highly qualified status, and the participation of 
master teachers to serve as instructors. 

Idaho funded four MSP projects in year one, all of 
which focused on mathematics. The total number of 
teachers served was 163. Two of the projects served 
schools in northern Idaho and two were based in 
southern Idaho. Although the Idaho projects all 
focus on mathematics, the state did not exclude sci- 
ence projects from the Idaho MSP program. 

There were six MSP projects funded in Montana. 
Three projects focused on mathematics, two on 
science, and one on both. The projects served 165 
teachers in the first year. Three projects served 
teachers in western Montana, one project was 
based in eastern Montana, and two projects served 
teachers from both sides of the state. 

In year one, Oregon funded four projects. Three 
projects were focused on mathematics and one 
on science. The projects served approximately 142 
teachers. Two projects were based in northwest- 
ern Oregon, one in the central region of the state, 
and one in the eastern region. The Oregon MSP 
program emphasized K-12 mathematics, although 
it did not exclude science-focused projects from 
eligibility. The state required a two-week institute 
with follow-up during the year. Unlike other states 
in the region, Oregon gave preference to projects 
that used an experimental or quasi-experimental 
design for the evaluation. Due to these challeng- 
ing requirements, the Oregon MSP program also 
recommended that the projects limit the number 
of teachers served. 

Washington funded four projects in year one. Two 
projects integrated mathematics and science, one 
project treated both subjects separately, and one 
project focused only on mathematics. The partner- 
ships served 258 teachers and were distributed 
across the state— two were based in western 
Washington, one in central Washington, and one 
in the eastern region of the state. 

The Washington MSP program required a two- 
week summer institute with follow-up. The state 
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also tailored the requirements to emphasize Wash- 
ington priorities for mathematics and science, 
placing emphasis on pedagogical content knowl- 
edge and teacher collaboration. 



The Title IIB program has multiple implications for 
analyzing how policy is translated into professional 
development program implementation 

The Title II legislation focuses heavily on policy- 
making at the federal and state levels around the 
quality of the teaching workforce (Plecki & Loeb, 
2004). Many of the policy issues in Title II focus 
at the level of the classroom teacher, including re- 
cruitment, induction, retention, certification, and 
compensation. However, the Northwest Region 
states have established Highly Objective Uniform 
State Standards for Evaluation, as required by the 
No Child Left Behind Act, that have resulted in 
most veteran teachers already being identified as 
highly qualified. 

One policy issue is what states are doing to provide 
support for teachers’ work. As the primary vehicle 
for professional development in science and math- 
ematics education, the Title IIB MSP program is a 
key leverage point in state education policy, deter- 
mining what is sanctioned as effective professional 
development. Title IIB MSP projects represent one 
way that states are addressing the issue of highly 
qualified teachers. For the most part. Title IIB 
MSP resources are used largely to improve the 
disciplinary knowledge and teaching skills of in- 
service teachers, who are typically already licensed 
to teach. This policy issue is the focus of the first 
research question articulated below. 

A second policy issue is the nature of accept- 
able evidence of the effectiveness of professional 
development. This includes a project evaluation 
that addresses measurable objectives of improve- 
ment of teacher mathematics and science content 
knowledge and pedagogical skills and of increases 
in student achievement. The U.S. Department of 
Education’s Strategic Plan 2002-2007 provided 
criteria to “transform Education into an evidence- 
based field” (U.S. Department of Education, 2002, 



p. 53). The Education Science Reform Act of 2002 
emphasizes the role of experimental and quasi-ex- 
perimental research and evaluation to determine 
the effectiveness of educational programs. This 
policy issue is the focus of the second and third 
research question articulated below. 

The purpose of the analysis in this report is to 
provide a description of the professional develop- 
ment and evaluations conducted in the first year 
of the Title IIB MSP programs in the Northwest 
Region. The analysis is not intended to provide 
information about the effectiveness of the profes- 
sional development or the evaluations. Such an 
analysis would require more extensive sources of 
data and a direct assessment of the projects, which 
are beyond the scope of this project. 



WHAT IS THE NATURE OF THE PROFESSIONAL 
DEVELOPMENT PROVIDED BY THE TITLE IIB 
MSP PROJECTS IN THE NORTHWEST REGION? 

This section summarizes the findings from the 
descriptive analysis and provides an overview of 
the professional development conducted in year 
one. The nature of the professional development 
being implemented in each of the projects was 
examined using an analytic framework based 
on the National Evaluation of the Eisenhower 
Professional Development Program (Desimone et 
al, 2002; Caret, Birman et al., 1999; Caret, Porter 
et al, 2001; Porter et al, 2000). Full descriptions 
of each project can be found in appendix A. The 
framework is organized around six features of 
high quality professional development: duration, 
activity type, collective participation, content 
focus, active learning, and coherence. (The analy- 
sis framework for professional development is 
included in appendix D). 

In conducting the descriptive analysis of the 
professional development, the proposals, year one 
evaluation reports, and interviews were examined 
to identify evidence related to each of the criteria 
and subcriteria in the analysis framework. The 
data were organized into an analysis framework 
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matrix and then summarized into the project 
tables that are included in appendix A. The de- 
scriptions should not be read as an evaluation of 
the effectiveness of the projects. 

The summary of the qualitative analysis that fol- 
lows was developed to look at patterns in profes- 
sional development provided by the Title IIB MSP 
projects in the Northwest Region. The intent of the 
summaries is to provide an overview of the infor- 
mation available in the full descriptions.^ 

To create this summary, the authors iteratively as- 
signed projects to categories based on the available 
information related to the dimensions of the analysis 
framework. The categories that were used to sum- 
marize the qualitative analysis included the extent 
to which the criteria were addressed, or reported. 

For example, a component of a project was assigned 
to the “Addressed” category if information related 
to the dimension was explicitly put forth in both the 
proposal and the annual report. The “Somewhat ad- 
dressed” category was assigned if the criterion was 
treated minimally or the information provided was 
incomplete. For example, some of the proposals and 
annual reports included statements that the project 
was aligned with standards, but no specific informa- 
tion was provided. The category “Not addressed” 
indicates that there was evidence that the project 
did not include the aspect of professional develop- 
ment or evaluation, while “Not reported” indicates 
that the documentation does not provide sufficient 
information to assign the project to a category. 



Duration provides opportunities for in- 
depth study and ongoing support 

Duration includes the total number of contact 
hours and the span of the program. Duration is 
related to the quality of professional development 
in at least two ways (Caret, Birman, et al., 1999). 
First, longer activities provide opportunities for in- 
depth study of content and pedagogy. In addition, 
activities that take place over a substantial span of 
time provide more opportunities to support teach- 
ers in trying out new practices. Exemplary profes- 
sional development programs generally provide 



FIGURE 1 

Most projects provided 80 or more hours of 
professional development 

■ Addressed 

■ Somewhat 
addressed 

■ Not addressed 



Duration 




Number of projects 



learning experiences that are at least 80 hours long 
(Caret, Birman, et al. 1999). 

In year one 13 projects provided 80 or more hours 
of professional development and were assigned to 
the “Addressed” category (figure 1). One project 
was assigned to the “Somewhat addressed” cat- 
egory because evidence of duration was reported 
in terms of weeks rather than hours or days. The 
remaining five projects did not provide sustained 
professional development. 



Activity type has two dimensions: traditional and reform 

Activity type has two dimensions. Traditional 
activities include within-district workshops and 
conferences, courses for college credit, and out- 
of-district workshops and conferences. Reform 
activities include teacher study groups; teacher 
collaboratives, networks, and committees; mentor- 
ing; internships; and resource centers. Reform ac- 
tivities often take place during the school day and 
may be more consistent with teachers’ goals and 
other professional development activities (Caret, 
Birman, et al., 1999). 

Two projects were assigned to the “Addressed” 
category because their primary means of provid- 
ing professional development was reform activi- 
ties (figure 2). Nine projects were assigned to the 
“Somewhat addressed” category because they 
provided a mix of traditional and reform activities. 
The eight projects in the “Not addressed” category 
conducted only traditional activities. One proj- 
ect was included in the “Not reported” category 
because there was not enough information to 
characterize the activities. 
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FIGURE 2 

More than half of projects included at least some 
reform activities 



FIGURES 

Almost two-thirds of projects included collaboration 
among teachers from the same school or district 
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Collective participation includes activities for 
teachers from the same school or district 

Collective participation includes activities de- 
signed for groups of teachers from the same school 
or district as opposed to individual teachers. Ac- 
tivities that feature collective participation provide 
opportunities for teachers to support each other as 
they attempt to incorporate new knowledge and 
practices (Caret, Birman, et al, 1999). Collective 
participation may also make it more likely that 
structural or organizational support will be put in 
place. 

Twelve projects supported collaboration among 
teachers from the same school or district and were 
thus included under the “Addressed” category 
(figure 3). The remaining seven projects were not 
designed to explicitly support collective par- 
ticipation, although many of them did provide 
opportunities for participants to collaborate 
with each other. While such opportunities may 
be valuable, they do not address the same issues 
that are reflected in the criterion for collective 
participation — fostering schoolwide or dis- 
trictwide support for teacher growth and im- 
proved instruction. 



dimensions: content knowledge, methods of 
teaching specific content, and emphasis on how 
students learn specific content (Caret, Birman, 
et al., 1999; Caret, Porter, et al, 2001). 

Projects were assigned to categories for each of the 
dimensions — some projects addressed multiple as- 
pects of content focus while others addressed only 
one. Projects that appeared to have a less focused 
approach — indicated by coverage of multiple topic 
areas or lack of detail about the content— were as- 
signed to the “Somewhat addressed” category. Two 
projects in the “Not addressed” category did not 
identify a focus for the professional development, 
while one project did not provide any information 
about the content of the professional development. 
More projects fell in the “Addressed” category for 
emphasis on content knowledge and teaching spe- 
cific content than for how students learn content 
(figure 4). 

In interviews project staff identified a tension 
between having enough time to address both 
content and how to incorporate such content into 
instructional units. However, many project staff 
also indicated that two weeks was the limit for 
engaging teachers. 



Content focus addresses the substance 
of professional development 

Content focus is the degree of emphasis on deep- 
ening teachers’ content knowledge in mathematics 
and science. There is evidence that professional 
development focused on content is related to 
increased student achievement (Cohen & Hill, 
1998; Kennedy, 1998). Content focus has three 



FIGURE 4 

Most projects at least somewhat addressed content 
knowledge and teaching specific content 
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Active learning describes participants' 
learning experiences 

Active learning includes opportunities for teachers 
to use new knowledge and practices with support 
and feedback. It is divided into four dimensions 
to provide a more specific description of partici- 
pants’ learning experiences: observing and being 
observed, planning for classroom implementa- 
tion, analyzing student work, and presenting and 
leading discussions and writing reports or plans 
(Caret, Birman, et al, 1999). 

All projects provided some form of active learn- 
ing except for three that did not provide evidence 
related to the format of the learning opportunities 
and were assigned to the category “Not reported.” 
Projects that provided evidence of some types of 
active learning but not others were placed in the 
“Not addressed” category for the dimensions not 
included in their activities. Projects that provided 
opportunities for participants to observe modeled 
instruction but not to be observed and receive 
feedback were categorized as “Somewhat ad- 
dressed” for the observing and being observed di- 
mension (figure 5); one project provided opportu- 
nities for a small subset of participants to present 
and was categorized as “Somewhat addressed” for 
the presenting and leading dimension. 

The most common form of active learning in the 
Northwest Region MSP projects was planning 
for implementation, which was included in 15 
projects. Nine projects provided opportunities for 
participants to observe instruction, to be observed, 
or both. Significantly fewer projects conducted 
activities in which participants had opportunities 
to make presentations or to analyze student work. 

In interviews disciplinary faculty indicated that 
integrating professional development into the 
work of classroom teachers was a challenge. Lack 
of sufficient funds to support ongoing interactions 
with faculty in institutions of higher education 
and the difficulty integrating service work with 
schools into tenure requirements were two prob- 
lems noted. 



FIGURES 

Most projects provided active learning in the form of 
planning for implementation 
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Coherence indicates how connected 
professional development is to other teacher 
learning and development activities 

Coherence is the degree to which the professional 
development is part of a wider set of activities for 
teacher learning and development. It includes con- 
nection to other professional development activi- 
ties and to teachers’ professional goals, alignment 
with state and district standards and assessments, 
and support for sustained communication among 
teachers (Caret, Birman, et al., 1999). Coherence 
was related to improvements in knowledge and 
practice reported by teachers (Desimone et al, 
2002; Caret, Porter, et al, 2001). 

Eight projects provided evidence that the activi- 
ties were connected to teacher needs or to other 
professional development and were assigned to 
the “Addressed” category (figure 6). One project 
reported providing information that could help 
create coherence in this dimension and was placed 
in the “Somewhat addressed” category. The other 
10 projects did not provide any information on 
this dimension. 

All of the projects provided some evidence that the 
activities were aligned with standards, but six proj- 
ects provided more detailed information and were 
assigned to the “Addressed” category. Ongoing 
communication was more common than connect- 
ing the project to teachers’ needs or other profes- 
sional development efforts. Projects were assigned 
the category “Somewhat addressed” if issues of 
ongoing communication were identified, but it was 
not clear how the issues would be addressed. 
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FIGURE 6 

Ongoing communication was the most common 
dimension of coherence addressed by projects 
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development of the partnership. An additional 
precondition is the extent to which the needs of 
the partners are congruent and the partnership 
enables them to pursue mutual goals. Just over 
half the projects included information about 
prior collaboration (figure 7). Only three projects 
included goals that were mutually beneficial to all 
partners — most of the activities were designed to 
produce outcomes only for teachers and schools. 



There are four criteria for describing partnership 



Complexity characterizes the structure of the partnership 



The analysis of the professional development also 
includes criteria that examine the nature of the 
partnerships that are demonstrated in project 
documents and through interviews. It has been 
more problematic to structure the criteria for what 
constitutes an effective partnership. The work of 
Gordon Kingsley and his group at the Georgia 
Institute of Technology has informed the develop- 
ment of the characteristics to describe “partner- 
ship” in the Title IIB MSP projects. A partnership, 
for the purpose of this analysis, is a group of enti- 
ties (organizations such as schools, colleges or uni- 
versities, and for-profit or non-profit companies) 
that work together to accomplish a set of mutual 
goals. The four dimensions for describing partner- 
ship include supporting preconditions, complexity, 
interdependence, and communication (Kingsley & 
O’Neil, 2004; Kingsley & Waschak, 2005). 

For the summary of partnership descriptions, the 
projects were divided into only two categories. The 
“Included” category indicates that the documenta- 
tion included information related to that dimension. 
The “Not included” category indicates that the proj- 
ects did not provide information. This binary system 
was used because the available evidence related to 
partnership categories was very limited. In addition, 
many of the dimensions related to partnership are 
not indicators of quality but are merely descriptive. 



Preconditions for partnership include 
existing relationships and mutual goals 

Partnership preconditions include existing 
relationships between organizations prior to the 



Complexity encompasses several dimensions to 
describe the structure of a partnership. A partner- 
ship with a vertical structure is hierarchical, and 
a partnership with a horizontal structure includes 
peer organizations on the same level. Partnerships 
with sector complexity include organizations with 
different areas of work. Spatial complexity refers 
to size of the geographic area that the partnership 
serves. 



The MSP projects were characterized as having 
either a vertical or horizontal structure, with more 
projects falling into the vertical category than the 
horizontal category (figure 8). These categories are 
merely descriptive — one type of structure is not 

FIGURE? 

Most activities were designed to produce outcomes 
only for teachers and schools 
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FIGURES 

More projects had a vertical structure than had a 
horizontal structure 
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considered more desirable than the other. Only 
one project involved partners from sectors other 
than education (figure 9). On the other hand, 10 
projects were spatially complex and served a large 
geographic region. 



Interdependence describes how the 
partners organize their work 



Interdependence characterizes the extent to which 
partners depend on each other for resources or 
materials and how they accomplish their work. 
Partners with reciprocal interdependence share 
their work back and forth. With sequential in- 
terdependence, the work of one partner will feed 
into the work of another. Pooled interdependence 
characterizes partnerships in which the members 
work independently for the most part. 



None of the projects indicated that the partners 
had a sequential approach to interdependence 
(figure 10). This is likely the result of the available 
documentation, which provided little detail about 
how the projects intended to go about partner 
collaboration. More projects fall into the pooled 
category, in which the partners have different re- 
sponsibilities, than into the reciprocal category, in 



FIGURE 9 

More than half the projects served a large geographic 
region 
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which the partners share responsibilities. Some of 
the projects used a mixture of pooled and recipro- 
cal collaboration. 



Communication describes the frequency 
of contact among partners 

The final partnership dimension is communica- 
tion. The characteristics of communication in the 
projects were more difficult to identify than those 
in the other dimensions, and the inferences made 
are more tentative. Communication refers only to 
the frequency of communication among the part- 
ners rather than the quality or direction of com- 
munication. Just over half of the projects provided 
information about frequency of communication in 
the documentation (figure 11). 



Only a brief comparison with the Council of Chief 
State School Officers project is possible 

At the time this report was written, CCSSO has 
not released a full-scale report of its study. Im- 
proving Evaluation of Professional Development in 
Mathematics and Science Education. As a result, 
only a brief comparison between the two projects 
is possible. Some of the general findings from 
the CCSSO study include a consistent focus on 
content knowledge (12 of 15 projects) and align- 
ment with standards (13 of 15 projects). More of 
the projects in the Northwest Regional Educa- 
tional Laboratory study provided active learning 
opportunities via planning for implementation 
(15 of 19 projects) and analyzing student work (6 
of 19 projects) than the CCSSO projects did— 7 of 
15 projects for planning and 3 of 15 projects for 
student work. 



FIGURE 10 FIGURE 11 

A pooled approach to collaboration was more Just over half of projects provided information about 

common than a reciprocal approach frequency of communication in the documentation 
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WHAT IS THE NATURE OF THE EVALUATIONS 
OF THE TITLE MB MSP PROJECTS IN 
THE NORTHWEST REGION? 

The second research question examines the nature 
of the evaluations of the Title IIB MSP projects 
and the extent to which the project addressed 
standards for the design and implementation of 
program evaluations. The authors constructed 
descriptions of the nature of the evaluation in each 
project using evidence from the project documents 
and interviews conducted at the end of year one. 
The data were organized into a project evaluation 
analysis framework matrix and then summarized 
into project tables (see appendix B). 

The summary of the qualitative analysis that 
follows was developed to look at patterns in the 
evaluations provided by the Title IIB MSP projects 
in the Northwest Region. The intent of the summa- 
ries is to provide an overview of the information 
available in the full descriptions. The descriptions 
of the project evaluation should not be read as an 
evaluation of the effectiveness of the projects. The 
analysis framework for project evaluation can be 
found in appendix D.^ 

To create the summaries of evaluation design and 
implementation, the authors iteratively assigned 
project evaluation features to categories based on 
the available information related to the dimen- 
sions of the evaluation analysis framework. A fea- 
ture of a project was assigned to the “Addressed” 
category if evidence related to the dimension 
was explicitly put forth in the proposal, the set 
of project interviews, and the annual report. The 
“Somewhat addressed” category was assigned if 
the criterion was treated minimally or the infor- 
mation provided was incomplete. For example, 
in many of the proposals and annual reports, 
the evaluator was identified, but no information 
was provided about that person s experience to 
establish credibility. The category “Not addressed” 
indicates that there was evidence that the project 
did not include the aspect of professional develop- 
ment or evaluation, while “Not reported” indicates 
that the documentation does not provide sufficient 



information to assign the project to a category. 
More detail about how categories were assigned is 
provided in the summary sections for professional 
development and evaluation in the main report. 



Project context provides information about 
factors that might influence implementation 

A key element of an evaluation should be sufficient 
information about the project so that the evalua- 
tion audience can make inferences about factors 
that might contribute to project implementation 
and impact. This criterion includes a description of 
the project goals, objectives, and activities and an 
identification of appropriate stakeholders. Evalua- 
tions should also include some information about 
contextual factors that might have influenced the 
project. 

All the projects included some information about 
the project and identified the appropriate stake- 
holders in their proposals or annual reports. 
However, only eight projects included information 
about the project context that might have influ- 
enced project implementation (figure 12). 



Evaluation purpose should be clearly identified 

A second criterion of evaluation quality addresses 
the extent to which the evaluations of the projects 
identified ways in which information was used. For- 
mative evaluation information should be reported 
by the extent to which it was used by project staff to 
improve the professional development. Summative 
evaluation information communicates the extent to 
which project goals and objectives have been met. 

FIGURE 12 

Less than half of projects included information 
about the project context that might have influenced 
project implementation 
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A majority of the projects included at least some 
identification about the purpose of the project, 
and formative purposes were most frequently cited 
(figure 13). Only four project evaluations included 
details about how the projects used the infor- 
mation from the evaluation to improve project 
function. Almost one-third of the projects did not 
indicate the purposes the evaluation served. 



Evaluation questions should be aligned with project goals 



FIGURE 14 

Few projects had evaluation questions that were 
clearly articulated, well aligned with project goals 
and objectives, and explicitly tied to data sources 
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A key feature of a quality evaluation is the identifi- 
cation of evaluation questions and their alignment 
with project goals and objectives. In addition, 
evaluation questions should indicate how evalu- 
ation information might be used by appropriate 
stakeholders. 

Only two projects had evaluation questions that were 
clearly articulated, well aligned with project goals 
and objectives, and explicitly tied to data sources 
(figure 14). Another five projects stated evaluation 
questions, but the questions were either not aligned 
with project goals or not connected to sources of in- 
formation to answer them. In the remaining projects 
evaluation questions were either not stated simply 
repeated the project goals or objectives. 



The identity and credibility of the 
evaluator should be clear 



Seven projects identified a specific person for the 
evaluation and included information about the 
evaluator s qualifications (figure 15). In eight proj- 
ects the evaluator was identified, but no additional 
information about the evaluator was included. In 
two projects the evaluator identified in the pro- 
posal was not used for the evaluation, and in two 
others the evaluator was not identified at all. 



Evaluation plans should describe stakeholder involvement 

An evaluation should describe how the positions 
and perspectives of the stakeholders will be taken 
into account throughout the evaluation. A descrip- 
tion of stakeholders’ involvement in the evalua- 
tion illustrates how they influence the evaluator’s 
understanding of project goals and objectives, how 
evaluation questions are shaped, and how results 
are reviewed. 



The evaluator of a project should be clearly identi- 
fied and the relationship between the evaluator 
and project should be articulated. In addition, the 
professional qualifications and experiences of the 
evaluator should be briefly described to establish 
one element of trust in the findings. 

FIGURE 13 

Few projects included details on how the projects 
used the information from the evaluation 



Only seven project evaluations mentioned stake- 
holder involvement with the evaluation (figure 16). 
Involvement was defined as having contributed 
information to the evaluation, and formative use 
of evaluation information was largely limited to 

FIGURE 15 

More than a third of projects identified a specific 
person for the evaluation and included that person's 
qualifications 
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project staff responsible for the development of 
professional development activities. In the remain- 
ing projects no indication of stakeholder involve- 
ment was provided. 



Methodological approaches need to be clearly described 



The evaluation plan should describe the proposed 
methodological approaches. The description 
should provide information about how the evalu- 
ator will identify and use data that help answer 
the evaluation questions within the constraints 
of time and cost. The data gathered need to be 
aligned with the goals that the project is intended 
to achieve. 



Four projects included sufficient information 
about the design of the evaluation to identify 
the methodological approach (figure 17). They 
attempted some form of experimental or com- 
parison group design, with enough information 
to explain how the design was implemented. In 
seven projects a methodological approach was 
initially identified and included both quantitative 
and qualitative evaluation designs, but there was 
insufficient evidence in the annual report that the 
approach had actually been implemented. Two 



FIGURE 16 

Seven project evaluations mentioned stakeholder 
involvement 
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projects indicated in their proposals that they 
would use a comparison group approach but did 
not implement such a design. The evaluations that 
used a qualitative evaluation approach included 
limited information about the design elements. 
Almost half of the projects did not indicate any 
methodological approach. 



Plans should report information sources and sampling 

The sources of information that will be used in the 
evaluation should be described in enough detail 
to assure that the information will be sufficient to 
meet the evaluation s purposes. Information about 
how the groups were selected to provide informa- 
tion should be identified and briefly described. 

If stakeholders are sampled, sampling strategies 
need to be clearly described. 

Almost one-third of the project evaluations 
included some characterization of the sources of 
information for their evaluations as well as how 
those sources were chosen and how samples were 
determined (figure 18). Nine projects identified the 
sources of information. But four did not include 
any information about data sources or about who 
supplied comments about professional develop- 
ment activities, meaning the reader does not know 
how many participants commented in support of 
the project. 



Information about data collection instruments is essential 

The evaluation should describe the nature of the 
data collection instruments and how they will be 
used to gather needed information. Information 
should also be included that demonstrates how 



FIGURE 17 FIGURE 18 

Over half the projects included some information More than half of projects identified the sources of 

about the design of the evaluation information 
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instruments are used to examine evidence of data 
reliability and validity. 

More than one-third of the projects specifically 
identified the instruments used to collect data 
and included those instruments as appendixes 
in their annual reports, but they did not provide 
information about quality control characteristics 
of the use of the instruments, such as evidence of 
reliability and validity (figure 19). An additional 
10 projects indicated that they used instruments, 
frequently project designed, but did not include 
the instruments in their reports. Two projects did 
not specify instruments in enough detail to inform 
the reader of what data were collected. 



Specific data collection procedures should be identified 

The evaluation report should describe how and 
when data were collected from sources. This 
information should include how the sources will 
answer evaluation questions through triangula- 
tion and multiple perspectives. The description 
of the data collection and its intent will provide a 
context for the eventual judging and interpreting 
of evaluation findings and recommendations. 

Only four projects provided specific information 
about when and how data were collected, but it 
was unclear who was responsible for collecting the 
data (figure 20). A majority of projects indicated 
that data collection occurred but provided no 
specific information about when and how. Almost 
one-quarter of the projects did not address when 
and how data were collected. 



FIGURE 19 

Most projects provided some information about 
instruments they used to collect data for their reports 
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FIGURE 20 

Few projects provided complete information on when 
and how data were collected 
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Descriptions of quantitative analysis 
processes should be detailed 

Evaluations need to show the nature and appropri- 
ateness of the quantitative analysis procedures and 
their relationships to the evaluation questions and 
data sources. Information about how the evalu- 
ation addresses the practical significance (e.g., 
effect sizes) and replicability, as well as statistical 
significance, should be included. 

Only three projects included information about 
which quantitative analysis procedures were used 
to conduct statistical analysis of project data (figure 
21). One-quarter indicated using descriptive statis- 
tics to report survey results, and another quarter 
reported statistically significant results but provided 
no information about how their analyses were per- 
formed. Reporting significance without providing 
transparent data and analysis processes is a major 
flaw in these evaluations. The remaining six projects 
provided no indication of quantitative analysis. 



Qualitative analysis processes should be articulated 

Evaluations need to show the nature and appropri- 
ateness of the qualitative analysis procedures and 

FIGURE 21 

Most evaluations did not provide complete 
information on quantitative analysis 
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FIGURE 22 

Only one project provided any information about 
how qualitative data were analyzed 



Qualitative 

analysis 




Number of projects 



■ Addressed 

■ Not addressed 
B Not reported 



successful or appropriate or successful for the Title 
IIB MSP projects in the Northwest Region. No 
project conditions supported successful imple- 
mentation of an experimental or quasi-experimen- 
tal evaluation design in the Title IIB MSP projects. 
But the extent to which such rigorous evaluation 
models are appropriate for Title IIB MSP projects 
is less straightforward. 



their relationships to the evaluation questions and 
data sources. The extent to which the findings are 
supported by qualitative data gathered from more 
than one source should be addressed. 

Only one project provided at least some informa- 
tion about how qualitative data were analyzed 
and addressed triangulation methods (figure 22). 
Most projects indicated that qualitative data were 
analyzed but failed to provide sufficient informa- 
tion as to how. And two projects did not report any 
qualitative data. 



Results from the evaluation analyses here 
cannot be compared with those from the 
Council of Chief State School Officers 

The results of the evaluation analyses from this 
Fast Response Task and the CCSSO study are not 
comparable because two different sets of criteria 
were used. The CCSSO study rated the evalua- 
tions according to Guskey s model for evaluating 
professional development (Guskey 2000), while 
the Northwest Regional Educational Laboratory 
study used criteria based on the Joint Commit- 
tee on Standards for Educational Evaluation 
(JCSEE 1994). 



UNDER WHAT CONDITIONS IS THE 
DEVELOPMENT OF EXPERIMENTAL OR QUASI- 
EXPERIMENTAL MODELS OF EVALUATION 
APPROPRIATE AND SUCCESSFUL? 

The third research question of this East Response 
Task asks under which conditions experimental 
or quasi-experimental evaluation designs are 



Only 7 of 19 funded projects included some evalu- 
ation design that included a control or comparison 
group. Two projects used randomized control 
experiments with random assignment of schools 
to a treatment and control group. Characterizing 
the other project evaluation designs in a quasi- 
experimental category is problematic because the 
evaluators did not provide sufficient indication of 
how they compared treatment and comparison 
groups These projects are thus referred to here 
as comparison group evaluation designs. Of the 
five projects with comparison group evaluation de- 
signs, two included the comparison group design 
in the proposal, but there was no evidence of the 
results in the year one report. Two other proj- 
ects used a comparison group design but lacked 
enough detail regarding the identification of the 
comparison group to determine the utility of the 
information provided. A fifth project allowed for 
self-selection of teachers into a treatment or com- 
parison group and provided limited information 
about the comparison group. 

The first issue that arises in the analysis is why 
only 7 of 19 funded projects even attempted a 
comparison group, quasi-experimental design, 
or experimental evaluation design. In interviews 
project staff and evaluators reported several issues 
that arose as they initially examined the potential 
to design and implement rigorous evaluations. 

The primary reason that projects did not seek to 
implement an experimental, quasi-experimental, 
or comparison group design was the difficulty 
or reported impossibility of recruiting a control 
or comparison group within the context of the 
professional development partnership. Project 
staff indicated that schools and districts refused 
to consider such assignment. Even when a project 
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sought to recruit a comparison group, they had 
difficulty matching the treatment and comparison 
groups at the grade level when their projects ad- 
dressed a broad span of teachers. Another reason 
was the lack of availability of adequate instrumen- 
tation. Project staff indicated that even if they had 
negotiated an agreement for a randomized design, 
they did not have adequate measures of changes 
in teacher content knowledge and instructional 
practices or of student achievement in project- 
focused content. They indicated that they did not 
consider the state assessments to be aligned with 
the content of the professional development. 

Adequate evaluation resources were identified as 
problematic by several projects. In many cases 
the evaluation budget was reported to be between 
$3,000 and $10,000. Conducting a rigorous evalua- 
tion for such a sum was cited as being impossible. 
Finally, evaluation capacity was indicated as a 
major reason why the models of evaluation were 
not rigorous. Few of the evaluators indicated that 
they had experience with or capacity to carry out a 
rigorous evaluation. 

Two of the projects attempted to implement an 
experimental design. In both the initial model of 
evaluation posited random assignment to treat- 
ment and control groups at the teacher level. 
However, both projects had to change that random 
assignment to a school level as teacher collabora- 
tion in either a formal lesson study approach or 
across grade levels through electronic support 
were an integral part of the professional develop- 
ment. Changing the unit of analysis from teacher 
to school resulted in a severe reduction in sample 
size. Project staff indicated that they were aware 
of the lack of power in their evaluation design 
and that this lack of power made determining an 
effect of the project on teacher content knowledge, 
instructional practices, or on student achievement 
very problematic. 

Both of the projects that did attempt more rigor- 
ous evaluations came from Oregon. The request 
for proposals in that state indicated that preference 
would be given to grant applicants who included 



randomized controlled trials or similar quasi- 
experimental methods in the project evaluations. 
In addition, the bidders sessions provided by the 
state education staff who were responsible for ex- 
plaining the grant program to potential applicants 
emphasized the need to conduct research within 
the Title IIB MSP program. This is not to say that 
simply exhorting projects to implement rigorous 
evaluations will result in projects engaging in such 
designs. However, the state education agency staff 
member indicated that including the increased 
preference for experimental or quasi-experimental 
evaluations was a major factor in the resulting 
attempts. 

In addition, the U.S. Department of Education has 
provided continued technical assistance to the 
state education agency staff and to projects on a 
regional basis. This technical assistance included 
information about assessments for both teachers 
and students that several project staff and evalu- 
ators indicated provided them with instruments 
that they might use to measure change in teacher 
and student content knowledge. State education 
agency staff have shared this information with 
project staff in periodic meetings throughout 
the past two years. They indicated that projects 
intended to increase the rigor of their evaluations 
in the second year of the projects. 



FINDINGS SUGGEST FUTURE 
DIRECTIONS FOR RESEARCH 

Two difficulties were encountered in the design of 
Title IIB MSP professional development programs. 
One is providing content to teachers and address- 
ing instructional needs. In interviews project staff 
identified tension between having enough time 
to address content and the ways in which such 
content should be incorporated into instructional 
units, especially within the two week window 
usually allotted for training workshops. An- 
other issue is the difficulty that project partners, 
especially disciplinary faculty, have integrating 
professional development into the work of the 
teacher in the classroom. Lack of sufficient funds 
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to support ongoing interactions between faculty in 
institutions of higher education and the difficulty 
integrating service work with schools into tenure 
requirements were two problems noted. 

The evaluation designs of Title IIB MSP programs 
are not rigorous enough to provide gold standard 
evidence of their effectiveness. Even evaluations 
that attempted an experimental or compari- 
son group design suffered from common flaws. 
Evaluations of many projects do not address key 
measurable outcomes or impacts of the projects. 
Eew projects used well developed instruments to 
measure changes in teacher content knowledge. In 
addition, the state assessment systems on which 
many of the projects relied were inadequate to 
directly measure the impact of projects on student 
achievement. 

Ongoing technical assistance is needed to increase 
the evaluation skills of the state education agency 
staff responsible for the Title IIB MSP programs 
and the staff and evaluators of the individual proj- 
ects. The U.S. Department of Education regional 
forums can be a start in informing stakeholders 
about the method and instruments to improve 
evaluations, but access to these regional forums is 
limited by project budget constraints. 

But this analysis is based on the first year of 
implementing the Title IIB MSP programs, when 
evaluation designs may not be fully mature. The 
data analyzed were static, showing what the proj- 
ects intended to do in the proposals and what they 
could demonstrate in the first year annual report. 
The single interview conducted with the staff and 
evaluator in each project captured information 
just after the second year of the summer institutes. 
State education agency staff indicated on several 
occasions that the projects had learned from 
their first year and that the projects’ professional 
development changed from year one to year two. 



In addition, the analysis was conducted by the two 
authors with assistance from a policy expert at the 
Northwest Regional Educational Laboratory. There 
has been no opportunity to include additional 
reviews to examine the data sources to refine the 
analyses. 

The Northwest Regional Educational Laboratory’s 
Center for Classroom Teaching and Learning has 
proposed extending this research to a second year 
of analysis of the Title IIB MSP projects. Year two 
annual reports will be collected and an expanded 
evaluation rubric will be constructed that exam- 
ines additional elements of the Joint Committee on 
Standards for Educational Evaluation that ad- 
dresses the criteria for the use of instruments and 
evaluation reports. In addition, this second year 
of the research would send the current analyses by 
state back to the state education agency staff and 
through them to the original project stakeholders 
for a member check. This is not seeking validation 
of the analysis. Rather, this member check would 
be an additional data source that would be ana- 
lyzed to determine how the Title IIB MSP is being 
manifested in the Northwest Region. 



1. Presenting this information in this com- 
pressed format means that there is a risk 
that the findings will be misrepresented. To 
understand what the summaries mean, it is 
necessary to interpret them in the context of 
the full project descriptions. 

2. Presenting this information here in a com- 
pressed format creates the risk that the 
findings will be misrepresented. To un- 
derstand what the summaries mean, they 
must be interpreted in the context of the full 
descriptions. 
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APPENDIX A 

ANALYSIS OF THE TITLE MB MATHEMATICS 
AND SCIENCE PARTNERSHIPS PROFESSIONAL 
DEVELOPMENT PROJECTS BY STATE 

This appendix addresses the first research ques- 
tion, which asks about the nature of the profes- 
sional development provided by the Title IIB 
MSP projects in the Northwest Region. Project 
documents and interviews were examined using 
the analysis framework to identify evidence of 
quality in the professional development. Evidence 
related to the nature of the partnerships was also 
examined. The descriptions should not be read as 
an evaluation of the effectiveness of the projects. 
Nor are they intended to compare the individual 
projects or the state programs to each other. 
Rather, the summaries describe the professional 
development funded through the program. In the 
following sections the evidence from each project 
is summarized and presented in tables. There are 
also summaries for each state and a final program- 
wide summary of the evidence related to each 
category of professional development. 

One overarching question that has emerged from 
conducting these analyses is the extent to which 
the projects must meet all criteria in every cat- 
egory in order to be considered effective. It is not 
clear whether professional development projects 
are more effective when they are rated highly in 
all categories than when they receive high ratings 
in only some categories. Nor is there evidence that 
any criterion is more important or less important 
than the others. Can a project that does not fully 
address one or more of the criteria still provide 
useful professional development for the teach- 
ers who participate? How many of the criteria for 
quality must be present in order for a project to 
be considered high quality? This issue cannot be 
resolved within the scope of the analysis of these 
programs included in this report. 



Alaska 

Alaska is the only state in the region that funded 
one project in the first year. The Alaska MSP 



program focused on K-8 mathematics. Some of 
the requirements included a summer institute of at 
least two weeks with follow-up during the school 
year, coursework to help teachers achieve highly 
qualified status, and the participation of master 
teachers to serve as instructors. 

The Journeys in Mathematics project served 51 
teachers in two school districts: Anchorage— a 
large urban district— and Lake and Peninsula— a 
large rural district with many remote schools. 

The partnership for this project included the two 
districts and the University of Alaska-Anchorage. 
The Journeys in Mathematics project activities 
included a three-week summer institute, three 
weekend workshops held during the school year, 
and collaborative work in lesson study teams and 
study groups (table Al). The content focus of the 
professional development was numeration. Partici- 
pants analyzed the content of mathematics lessons 
and explored how to teach mathematics to diverse 
students. 

Journeys in Mathematics provided sustained 
professional development that featured a balance 
of traditional and reform activities. There was a 
targeted focus on content knowledge and mul- 
tiple and varied opportunities for active learn- 
ing. While the project documentation referenced 
standards and participants used standards in their 
work, detailed information on specific standards 
addressed was not provided. The partnership 
featured spatial complexity, serving teachers in 
remote schools over a large geographic area. There 
was evidence of reciprocal interdependence among 
the partners and frequent communication. 



Idaho 

Idaho funded four MSP projects in year one, all of 
which focused on mathematics. The total number 
of teachers served was 163. Two of the projects 
served schools in northern Idaho, and two were 
based in southern Idaho. Although the Idaho 
projects all focus on mathematics, the state did 
not exclude science projects from the Idaho MSP 
program. 
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TABLE A1 

Journeys in Mathematics project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 140 hours of professional development 


Activity type 


Traditional — institutes, 
workshops 


Summer institute, weekend workshops 




Reform — school- or classroom- 
based activities 


Lesson study, study groups 


Collective 

participation 


Teachers from the same school 
or district work together 


Designed for teams of teachers from participating schools 


Content focus 


Content knowledge 


Targeted focus on numeration; specific topics identified (for 
example, operations, fractions, and decimals) 

Content faculty conducted professional development 




Teaching specific content 


Pedagogical focus on problem solving 

Participants analyzed lessons and explored how to teach 

mathematics to diverse students 




How students learn content 


No evidence identified 


Active learning 


Observing or being observed 


Participants observed model lessons and taught lessons in front of 
their peers 




Planning for classroom 
implementation 


Participants engaged in lesson study and analyzed classroom 
activities 




Reviewing student work 


Participants analyzed student work and identified implications for 
instruction 




Presenting leading discussions, 
writing 


Participants wrote reports and led study groups 


Coherence 


Connected to teachers' 
goals and other professional 
development 


Development of a new master's program in middle school 
mathematics education 




Aligned with standards and 
assessments 


Courses were designed to align with state and National Council of 
Teachers of Mathematics standards; participants used standards in 
their work 

No information about specific standards addressed 




Supports sustained 
communication 


Opportunities for collegial communication via lesson study, study 
groups, and other collaborative projects 


Partnership 


Existing relationships 


No evidence of prior collaboration among the partners 


preconditions 


Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


University responsible for leading the project; school districts 
responsible for recruiting, release time 




Horizontal 






Sector 






Spatial 


Project covered a large geographic area 


Interdependence 


Pooled 






Sequential 






Reciprocal 


Master teachers from the districts given a role in designing and 
providing the professional development 


Communication 




Project team met monthly during the spring and less frequently 
during the fall and winter 





20 



ANALYSIS OF TITLE MB MATHEMATICS AND SCIENCE PARTNERSHIPS IN THE NORTHWEST REGION 



Classroom Assessment for Mathematics Performance 

The Classroom Assessment for Mathematics Per- 
formance project partners included the University 
of Idaho in Moscow and six school districts in 
northern Idaho. The districts are not named in 
the documentation other than Lakeland, the lead 
local education agency. The project provided a 
two-week summer institute for 18 elementary and 
middle school teachers with follow-up conducted 
primarily via an online course. Teacher content 
knowledge was a secondary focus in the Class- 
room Assessment for Mathematics Performance 
project (table A2). The primary focus of the project 
was assessment. The summer institute focused on 
learning to use tools for organizing and analyzing 
data related to school math programs. The partici- 
pants analyzed data from state assessments, exam- 
ined student work for evidence of mathematical 
proficiency, and developed classroom assessments. 

The documentation from the Classroom Assessment 
for Mathematics Performance project did not articu- 
late the number of hours of professional develop- 
ment provided in year one. Other than the summer 
institute, it was not possible to determine whether 
the activities were traditional or reform. Although 
the focus of the project was on assessment, teachers’ 
content knowledge of number sense, as well as their 
knowledge of how to teach this content area, was 
addressed in the context of assessment. The profes- 
sional development was designed to meet the needs 
of the participating teachers. Although the project 
included a lead school district partner, additional 
districts were referenced, but not identified. 

Coeur d'Alene — Mathematics 

In year one, the Coeur d’Alene project served 34 
teachers from 12 schools who taught in grades one 
through nine. The partners included the Univer- 
sity of Idaho in Moscow, Coeur d’Alene School 
District, and Washington State University. The 
project provided a two-week summer institute 
and during the school year, teachers participated 
in six Saturday sessions— three in the fall term 
and three in the spring. The content focus of the 



project was number sense (table A3). The Coeur 
d’Alene project focused on providing teachers 
with conceptual understanding of the content that 
would improve their ability to provide effective 
instruction to students. 

The Coeur d’Alene MSP project provided a 
sustained learning experience for teachers that 
featured traditional forms of professional devel- 
opment. There was a targeted focus on content 
knowledge with opportunities for active learning 
provided in the form of planning for implementa- 
tion. The partnership had a horizontal structure 
and reciprocal interdependence — representatives 
from all partners were involved in designing the 
professional development. There was evidence of 
frequent communication among the partners. 

Developing Mathematical Thinking 

The Developing Mathematical Thinking project 
served 56 kindergarten through grade six teachers 
from four schools in three Boise-area districts. The 
partners included Boise State University; Boise, 
Caldwell, and Emmett school districts; and the Lee 
Pesky Learning Center. The Developing Math- 
ematical Thinking project began with a week-long 
institute during the summer. Project staff made 
monthly visits to each teacher’s classroom, and 
teacher teams met once a week. The first year of 
the project was focused on number (table A4). 
Participants investigated how students learn by 
reading research and by analyzing videos and 
examples of student work. 

The Developing Mathematical Thinking project 
provided sustained professional development 
that featured primarily reform activities in ad- 
dition to a traditional summer institute. There 
was a targeted focus on content knowledge and 
how students learn specific content with multiple 
and varied opportunities for active learning. The 
project documentation included detailed informa- 
tion on alignment with standards and participants 
used standards in their work. The partnership had 
a vertical structure and pooled interdependence, 
with the project primarily driven by the university. 
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TABLE A2 

Classroom Assessment for Mathematics Performance project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately four weeks of professional development, 
including work online 


Activity type 


Traditional — institutes, workshops 


Summer institute 




Reform — school- or classroom- 
based activities 


Other activities difficult to characterize without more 
information 


Collective 

participation 


Teachers from the same school or 
district work together 


Not designed to support collective participation 
Collaboration among individual participants 


Content focus 


Content knowledge 


Primary focus on assessment rather than content; secondary 
focus on number sense addressed in the context of assessment 
Content faculty involved in developing the professional 
development 




Teaching specific content 


Emphasis on how to teach content that is being assessed 




How students learn content 


Evidence of some attention to student learning via analysis of 
assessments 


Active learning 


Observing or being observed 






Planning for classroom 
implementation 


Participants developed an implementation plan and examined 
lessons 




Reviewing student work 


Participants analyzed student artifacts and developed 
assessment items 




Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 


Teacher needs were used to design the project activities 




Aligned with standards and 
assessments 


Participants used standards, assessments in their work 
No information about specific standards addressed 




Supports sustained communication 


Opportunities for collegial communication via online discussion 
forum 


Partnership 

preconditions 


Existing relationships 


Project based on long-standing relationship between university 
and school district partners 




Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


University responding to district needs 

Partner districts were not named in the proposal or the report 

with the exception of the lead school district 




Horizontal 






Sector 






Spatial 




Interdependence 


Pooled 


University responsible for most aspects of the professional 
development 




Sequential 






Reciprocal 




Communication 




Not addressed 



Twin Falls — Curriculum improvement and 
alignment in middle school mathematics 

This project was based in the Twin Falls School 
District and involved a partnership between 



the district and the College of Southern Idaho. 
The activities served 55 teachers in grades five 
through eight from nine schools in the district, 
The Twin Falls project conducted three types 
of activities, all of which took place during the 
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TABLE A3 

Coeur d'Alene project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 96-120 hours 


Activity type 


Traditional — institutes, workshops 


Summer institute, workshops 




Reform — school- or classroom- 
based activities 


No evidence identified 


Collective 

participation 


Teachers from the same school or 
district work together 


Project focused on one school district; participants worked in 
grade-level teams 


Content focus 


Content knowledge 


Targeted focus on number sense; specific topics identified (for 
example, fractions, proportional reasoning, whole number 
concepts, operations) 




Teaching specific content 


Pedagogical emphasis on how to increase student number 
sense by using multiple representations 




How students learn content 


No evidence identified 


Active learning 


Observing or being observed 






Planning for classroom 
implementation 


Participants planned and delivered lessons, and evaluated and 
reflected on the results 




Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 


No evidence identified 




Aligned with standards and 
assessments 


Statements that content and materials were consistent 
with state and National Council of Teachers of Mathematics 
standards; National Council of Teachers of Mathematics standard 
identified: representation 




Supports sustained communication 


Opportunities to participate in online discussions 


Partnership 


Existing relationships 


No evidence of prior collaboration among the partners 


Preconditions 


Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 






Horizontal 


Proposal explicit about collaborative decisionmaking 




Sector 






Spatial 




Interdependence 


Pooled 






Sequential 






Reciprocal 


District staff and university faculty worked together to design 
project activities 


Communication 




Partners conducted face-to-face meetings, also communicated 
via online discussion forums 



school year (table A5). Teachers participated 
in four day-long trainings focused on content 
strands; six half-day training sessions; and 10 
1.5-hour planning sessions before each new 
unit of instruction. The content workshops were 
intended to bridge gaps in teachers’ knowledge of 
algebra I concepts. The project was designed to 



support middle school teachers in implementing 
a new curriculum. 

The Twin Falls MSP provided fewer hours of 
professional development than other projects and 
did not conduct a two-week summer institute. The 
activities were traditional forms of professional 





APPENDIX A 



23 



TABLE A4 

Developing Mathematical Thinking project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 200 hours of professional development; amount 
of project time varied from teacher to teacher 


Activity type 


Traditional — institutes, workshops 


Summer institute 




Reform — school- or classroom- 
based activities 


Classroom-based activities, teacher teams 


Collective 

participation 


Teachers from the same school or 
district work together 


Project designed for grade-level teams from the same school 
Funds used to provide time for collaborative work 


Content focus 


Content knowledge 


Targeted focus on number sense; specific topics identified (for 
example, operations, place value, ratio and proportion, fractions, 
decimals, percentages) 

Content faculty helped to design the professional development 




Teaching specific content 


Evidence not identified 




How students learn content 


Emphasis on investigating how students learn 


Active learning 


Observing or being observed 


Classroom modeling and coaching, peer observation 




Planning for classroom 
implementation 


Participants engaged in collaborative lesson planning 




Reviewing student work 


Participants analyzed student work 




Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 






Aligned with standards and 
assessments 


Alignment with standards described in detail and participants 
used standards in their work 

Content focus was selected to address the content strands of the 
state test 




Supports sustained communication 


Teacher teams met weekly 


Partnership 

preconditions 


Existing relationships 


Project based on ongoing collaboration between school districts 
and university 




Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


Project led by university with school administrators in a 
supporting role 




Horizontal 






Sector 






Spatial 




Interdependence 


Pooled 


University responsible for almost all aspects of the project 




Sequential 






Reciprocal 




Communication 




Not addressed, although university faculty made weekly visits to 
school sites 



development. There was a focus on content knowl- 
edge, although specific topics were not articu- 
lated, as well as some emphasis on analyzing how 
students learn. Twin Falls provided multiple forms 
of active learning and documented the alignment 



with standards in detail. The partnership had a 
horizontal structure and was guided by a project 
advisory committee that met on a quarterly basis, 
The school district had responsibility for most 
aspects of the project. 
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TABLE A5 

Twin Falls project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 60 hours of professional development 


Activity type 


Traditional — institutes, workshops 


District-based trainings 




Reform — school- or classroom- 
based activities 




Collective 

participation 


Teachers from the same school or 
district work together 


Project focused on one school district 
Participants were organized into cadres by grade level 


Content focus 


Content knowledge 


Content area identified: Algebra 1; specific concepts and topics 
were not addressed 

Content faculty served as project advisors 




Teaching specific content 


No evidence identified 




How students learn content 


Participants analyzed how students solve problems, common 
misconceptions 


Active learning 


Observing or being observed 






Planning for classroom 
implementation 


Participants planned units and lessons collaboratively 




Reviewing student work 


Analyzing how students solve problems 




Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 






Aligned with standards and 
assessments 


Activities explicitly aligned with standards, organized around 
state assessment strands 




Supports sustained communication 


Opportunities for communication via teacher cadres 


Partnership 

preconditions 


Existing relationships 


No details about prior collaboration included in documentation; 
proposal identified an ongoing commitment between the 
college and the district 




Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 






Horizontal 


Project advisory committee included representatives from all 
stakeholders 




Sector 






Spatial 




Interdependence 


Pooled 


Pooled: school district responsible for almost all aspects of the 
project 




Sequential 






Reciprocal 




Communication 




Advisory committee met four times per year 



Montana 

Montana funded six MSP projects in year one. 
Three projects focused on mathematics, two 
on science, and one focused on both. The proj- 
ects served 165 teachers in the first year. Three 
projects served teachers in western Montana, one 
project was based in eastern Montana, and two 



projects served teachers from both sides of the 
state. 

Billings 

The partners in the Billings MSP project included 
Billings School District and Montana State 
University-Billings. The professional development 
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TABLE A6 

Billings project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 1 00 hours of professional development 


Activity type 


Traditional — institutes, workshops 


Summer institute, follow-up meetings 




Reform — school- or classroom- 
based activities 


Classroom coaching 


Collective 

participation 


Teachers from the same school or 
district work together 


Project focused primarily in one district 
Participants worked in teams 


Content focus 


Content knowledge 


Multiple content areas: probability and statistics, geometry, and 
proportional reasoning 

Content faculty with dual appointments in mathematics 
and mathematics education helped plan the professional 
development and served as instructors 




Teaching specific content 


Mathematics pedagogy focused on manipulatives and journals; 
not clear that strategies were tied to the specific content of the 
professional development 




How students learn content 


No evidence identified 


Active learning 


Observing or being observed 


Teachers participated in classroom modeling and coaching 




Planning for classroom 
implementation 


Teachers designed and reflected on lessons 




Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 


Meetings with principals held during the year to address 
coherence 




Aligned with standards and 
assessments 


Statements that project was designed around National Council 
of Teachers of Mathematics standards, specific standards not 
identified 




Supports sustained communication 


Opportunities for collegial communication via teacher teams 


Partnership 

preconditions 


Existing relationships 


Partnership was based on existing collaborations between 
district and university 




Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 






Horizontal 


Partners shared responsibility for planning and executing the 
project 




Sector 






Spatial 




Interdependence 


Pooled 






Sequential 






Reciprocal 


Partners worked together to design and deliver the activity 


Communication 




Regular meetings, communication among partners 



served 27 teachers in grades six through eight 
from eight middle schools and six elementary 
schools. Most of the participants taught in the 
Billings district, but five came from outlying dis- 
tricts and private schools. Teachers participated 
in a two-week summer institute, four meetings 



during the year to extend the focus on content 
knowledge, and at least three sessions of job-em 
bedded coaching (table A6). The project focused 
on three strands in middle school mathematics: 
probability and statistics, geometry, and propor- 
tional reasoning. 
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The Billings MSP project provided sustained profes- 
sional development that featured a balance of tradi- 
tional and reform activities. There was a broad focus 
on content knowledge and the information about 
the pedagogical focus was not detailed enough to 
determine if it was tied to teaching specific content. 
The project featured multiple forms of active learn- 
ing. Meetings were held with school principals to 
help ensure the coherence of the professional devel- 
opment. The partnership had a horizontal structure, 
with evidence of reciprocal interdependence among 
the partners and frequent communication. 

Creating Opportunities in Mathematics 

for Exemplary Teaching 

The partners in the Creating Opportunities in 
Mathematics for Exemplary Teaching project 
included seven school districts, the Montana 
Council of Teachers of Mathematics, the Montana 
Learning Center, Montana State University-Boze- 
man, and the University of Montana-Missoula. 

The project served 75 teachers — 25 each from 
the elementary, middle school, and high school 
levels. Four content areas were identified for the 
professional development: number sense, alge- 
braic reasoning, data and probability, and think- 
ing mathematically. Because of the timing of the 
awards, the summer institute was postponed 
until the second year of the project. The Creat- 
ing Opportunities in Mathematics for Exemplary 
Teaching project staff used the first year to recruit 
teachers, conduct school visits, and design the 
professional development (table A7). 

The first year of the Creating Opportunities in 
Mathematics for Exemplary Teaching project was 
a startup year, focused primarily on planning and 
recruiting participants. The proposed professional 
development will address multiple content areas, 
with a general pedagogical focus rather than one 
tied to issues of teaching specific content (table 
A8). It was not possible to determine opportunities 
for active learning in the available documenta- 
tion. The project was based on existing relation- 
ships and had a horizontal structure. The partners 
engaged in frequent communication. 



Flathead and Salish Kootenai College 

The Flathead and Salish Kootenai College proj- 
ect served 12 teachers from six schools on the 
Flathead reservation. The participants taught 
mathematics in grades 1-12, with at least half 
teaching multiple grade levels. In addition to the 
schools, the partners included Salish Kootenai 
College and the Northwest Regional Educational 
Laboratory. Teachers participated in a two -week 
summer institute and four workshops held during 
the school year. Participants also received three or 
four sessions of classroom coaching. The project 
focused on algebra with a pedagogical emphasis 
on teaching American Indian students and Eng- 
lish language learners. 

The Flathead and Salish Kootenai College project 
provided sustained professional development 
that featured a balance of traditional and reform 
activities. Although the project did not technically 
feature groups of teachers from the same districts, 
the participants worked in teams based on geo- 
graphic location. This enabled a form of collective 
participation for the teachers in these small, rural 
schools. There was a targeted focus on content 
knowledge and the project featured multiple forms 
of active learning. There was some evidence of 
reciprocal interdependence among the partners 
and frequent communication. 

High School Chemistry in Western Montana 

The project partners included the University of 
Montana-Missoula and four school districts in 
western Montana. The professional development 
served 12 high school science teachers. The project 
conducted a two -week summer institute, and there 
were four one-day site visits during the school 
year (table A9). Chemistry was the content focus, 
organized around the history of science, and the 
pedagogical focus was inquiry. 

The professional development provided in the 
High School Chemistry project was sustained, 
with primarily traditional activities in addition 
to site visits that can be characterized as reform 
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TABLE A7 

Creating Opportunities in Mathematics for Exemplary Teaching project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 10 hours for 80 percent of the participants 


Activity type 


Traditional — institutes, workshops 


Workshop and meeting 




Reform — school- or classroom- 
based activities 




Collective 

participation 


Teachers from the same school or 
district work together 


Not designed to support collective participation 
Participants encouraged to collaborate, interact online 


Content focus 


Content knowledge 


Multiple content areas: number sense, algebraic reasoning, data 
and probability, and thinking mathematically 
Project design team included mathematicians, mathematics 
educators, and statisticians 




Teaching specific content 


Pedagogical goal general rather than specific: using reflection to 
improve teaching 




How students learn content 


No evidence identified 


Active learning 


Observing or being observed 


Learning experiences not described 




Planning for classroom 
implementation 






Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 


Content of the professional development identified based on 
teacher needs 




Aligned with standards and 
assessments 


Statements that project was designed around Montana 
standards and National Council of Teachers of Mathematics 
standards, specific standards not identified 




Supports sustained communication 


Opportunities for participants to communicate online 


Partnership 

preconditions 


Existing relationships 


Project based on a history of collaboration between the 
Montana Council of Teachers of Mathematics and state 
universities 




Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


Leadership shared between the Montana Council of Teachers of 
Mathematics and Montana State University 




Horizontal 






Sector 






Spatial 


Project served small rural schools throughout the state 


Interdependence 


Pooled 


Management and execution of the project 




Sequential 






Reciprocal 


Design of the professional development 


Communication 




Monthly meetings, ongoing communication via e-mail 



based. The professional development was focused 
on chemistry with emphasis on how students 
learn. Active learning opportunities were pro- 
vided via developing curriculum units. There was 
explicit and detailed information about the align- 
ment with state standards. It is unclear whether 



there was a partnership involved in this project. 
Although high-need school districts were identi- 
fied in the proposal, the university was respon- 
sible for all aspects of the project and individual 
teachers were listed as partners in the annual 
report. 
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TABLE A8 

Flathead and SaMsh Kootenai College project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 100-120 hours of professional development 


Activity type 


Traditional — institutes, workshops 


Summer institute, workshops 




Reform — school- or classroom- 
based activities 


Classroom coaching 


Collective 

participation 


Teachers from the same school or 
district work together 


Professional learning teams were based on geographical 
location 


Content focus 


Content knowledge 


Targeted focus on algebra; specific topics identified (for 
example, pattern recognition, use of models and symbols to 
represent relationships, analysis of change) 

Content faculty member served as instructor 




Teaching specific content 


No evidence identified 




How students learn content 


Mathematics pedagogy focused on teaching American Indian 
students and English language learners 


Active learning 


Observing or being observed 


Classroom coaching and modeling 




Planning for classroom 
implementation 


Participants develop lessons and rubrics, worked with their 
curricula 




Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 






Aligned with standards and 
assessments 


Statements that project was aligned with standards, specific 

standards not identified 

Participants used standards in their work 




Supports sustained communication 


Opportunities for collegial communication via teacher teams 


Partnership 

preconditions 


Existing relationships 


Partnership based on ongoing relationships between college 
and the schools 




Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


College had primary responsibility for almost all aspects of the 
project 




Horizontal 






Sector 






Spatial 




Interdependence 


Pooled 


Pooled: partners had separate responsibilities, although all had a 
role in designing the project 

Two organizations within college worked collaboratively to plan 
and deliver the professional development 




Sequential 






Reciprocal 




Communication 




Quarterly meetings 



Improving Science Teaching 

The Improving Science Teaching project served 
19 teachers in grades K-12 from six schools in six 
districts. The project partners were the University 



of Montana-Missoula and the six school districts. 
The professional development included a two -week 
summer institute and additional one-day sessions 
during the school year — three in the fall and three 
in the spring (table AlO). The content focus of the 
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TABLE A9 

High School Chemistry project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 1 04 hours of professional development 


Activity type 


Traditional — institutes, workshops 


Summer institute 




Reform — school- or classroom- 
based activities 


Site visits 


Collective 


Teachers from the same school or 


Not designed to support collective participation 


participation 


district work together 


There was collaboration among individual participants 


Content focus 


Content knowledge 


Focus on chemistry; specific topics identified (for example, gas 
laws, atomic theory, periodic table) 

Content faculty conducted professional development 




Teaching specific content 


Pedagogical focus on inquiry-based instruction 




How students learn content 


Emphasis on how students learn science 


Active learning 


Observing or being observed 


Some participants received classroom coaching 




Planning for classroom 
implementation 


Developing curriculum units 




Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 






Aligned with standards and 


Activities explicitly aligned with standards 




assessments 






Supports sustained communication 


Goal to provide teachers with networking opportunities; not 
clear if the project supported ongoing communication 


Partnership 


Existing relationships 


No information on prior collaboration 


preconditions 


Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


No evidence of partnership 




Horizontal 






Sector 






Spatial 




Interdependence 


Pooled 


No evidence of partnership 




Sequential 






Reciprocal 




Communication 




No evidence of partnership 



professional development was watershed ecol- 
ogy with a pedagogical focus on teaching science 
through inquiry. 

The professional development provided in the Im- 
proving Science Teaching project was sustained, 
with primarily traditional activities in addition 
to fieldwork with scientists that can be character- 
ized as reform based. The project documentation 
identified watershed ecology as the focus, but there 



was no detailed information about the content. 
There was also a pedagogical focus on inquiry 
grounded in examining how students learn. There 
was no evidence of prior collaboration between the 
partners or that the organizations named as part- 
ners in the proposal were involved in developing 
the project. However, the project staff stated that 
the collaboration between the science faculty and 
the education faculty within the university was a 
positive outcome of the project. 
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TABLE A10 

Improving Science Teaching project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 128 hours of professional development 


Activity type 


Traditional — institutes, workshops 


Summer institute, workshops 




Reform — school- or classroom- 
based activities 


Fieldwork with scientists 


Collective 

participation 


Teachers from the same school or 
district work together 


Not designed to support collective participation 
There was collaboration among individual participants 


Content focus 


Content knowledge 


Content area identified: watershed ecology; not information on 
specific topics 

Content faculty involved in delivering professional development; 
opportunities to work with scientists and other specialists 




Teaching specific content 


Pedagogical focus on teaching science via inquiry 




How students learn content 


No evidence identified 


Active learning 


Observing or being observed 


Participants observed modeled lessons and taught lessons in 
front of their peers 




Planning for classroom 
implementation 


Teachers developed lesson plans 




Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 






Aligned with standards and 
assessments 


Statements that project was aligned with standards, specific 
standards not identified 




Supports sustained communication 




Partnership 


Existing relationships 


No evidence of prior collaboration 


preconditions 


Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


University responsible for all aspects of the project 




Horizontal 






Sector 






Spatial 




Interdependence 


Pooled 


Only role for school district was to provide assessment data 




Sequential 






Reciprocal 




Communication 




Not addressed 



Montana Science and Mathematics Consortium 

The project partners included Libby School District, 
the University of Montana-Missoula, and eight 
additional school districts. Montana Science and 
Mathematics Consortium served 20 teachers from 
13 schools in nine districts. Most of the participants 
taught at the elementary or middle school level, but 



there were also three high school teachers. In year 
one the Montana Science and Mathematics Con- 
sortium project deviated from the original plan due 
to the timing of the awards— a three-day institute 
was held during the summer instead of a two-week 
event, with an additional one-day follow-up during 
the school year (table All). The full scope of activi- 
ties was postponed until the second year. 
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TABLEAU 

Montana Science and Mathematics Consortium project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Four days of professional development 


Activity type 


Traditional — institutes, workshops 


Traditional — workshop 




Reform — school- or classroom- 
based activities 




Collective 


Teachers from the same school or 


Designed for teams from the same school or district 


participation 


district work together 




Content focus 


Content knowledge 


Data not available 




Teaching specific content 






How students learn content 




Active learning 


Observing or being observed 


Data not available 




Planning for classroom 
implementation 






Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 






Aligned with standards and 
assessments 


Standards identified, but not clear how they were addressed 




Supports sustained communication 




Partnership Pre- 


Existing relationships 


No evidence of prior collaboration 


conditions 


Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 






Horizontal 


No clear lead organization 




Sector 






Spatial 




Interdependence 


Pooled 


Partners had independent responsibilities 




Sequential 






Reciprocal 




Communication 




Not addressed 



Very little documentation was available for the Oregon 

Montana Science and Mathematics Consortium 
project. The proposal was vague about the nature 
of the professional development and the project 
did not follow the design that was proposed. The 
partners had independent rather than shared re- 
sponsibilities, with the higher education institute 
responsible for designing and delivering the pro- 
fessional development and the lead school district 
handling both evaluation and financial tasks. The 
role of the other school districts was to provide 
release time for teachers. 



In year one Oregon funded four projects. Three 
projects were focused on mathematics, and one 
was focused on science. The projects served 
approximately 142 teachers. Two projects were 
based in northwestern Oregon, one in the central 
region of the state, and one in the eastern region. 
The Oregon MSP program emphasized K-12 
mathematics, although it did not exclude science- 
focused projects from eligibility. The state required 
a two -week institute with follow-up during the 
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year. All projects were expected to include an 
educational service district as one of the partners. 
Unlike the other states in the region, Oregon gave 
preference to projects that used an experimental or 
quasi-experimental design for the evaluation. Due 
to these requirements, the Oregon MSP program 
recommended that the projects limit the number 
of teachers served. 

Greater Oregon Mathematics Partnership 

The Greater Oregon Mathematics Partnership 
project provided a summer institute for eight 
teachers who were expected to go back to their 
schools and provide professional development for 
a total of 77 additional teachers over the course 
of the school year. The content focus was alge- 
braic reasoning for grades four through eight 
and the project used an Annenberg course as its 
curriculum. The partners included the Region 
18 education service district. Eastern Oregon 
University, the International Society for Technol- 
ogy in Education, and nine school districts. The 
institute provided two weeks of learning experi- 
ences for the core cadre (table A12). The core 
cadre members went on to provide 10 two-hour 
sessions for other teachers at their schools during 
the year. The modeling of the instructor during 
the institute was intended to address issues of 
instruction. 

The Greater Oregon Mathematics Partnership 
project provided sustained professional devel- 
opment for a small group of participants via a 
training-of-trainers model. Most of the other 
participants received significantly fewer hours of 
service. The project had a targeted content focus 
with opportunities for active learning limited 
to the modeling of the instructor. However, the 
small group of core cadre participants had ad- 
ditional opportunities in the form of providing 
professional development for the other partici- 
pants. The partnership had a vertical structure 
with pooled interdependence. According to the 
documentation, the involvement of the higher 
education partner was limited to oversight of 
credits. 



High Desert 

The High Desert project served 30 participants 
who taught grades three through eight at 22 
schools in five districts. The activities emphasized 
both content knowledge and pedagogical content 
knowledge in mathematics. The project partners 
included Oregon State University, the High Desert 
Education Service District, the Northwest Re- 
gional Educational Laboratory, and the five school 
districts in central Oregon. The project provided 
two weeks of professional development during the 
summer and three terms of coursework during the 
school year (table A13). 

The High Desert MSP project provided a sustained 
learning experience for teachers that featured 
traditional forms of professional development. 

The project not only provided opportunities for 
collaboration, but also sought to develop partici- 
pants’ abilities in this area. The project addressed 
multiple content areas with a focus on standards 
and pedagogical content knowledge. The educa- 
tion and mathematics faculty members conducted 
an analysis to ensure that the professional devel- 
opment was focused on content knowledge and 
aligned with the National Council of Teachers of 
Mathematics standards. The partnership had a 
vertical structure with the university responsible 
for most aspects of the project. 

Northwest Regional Education Service District 

The activities of the Northwest Regional Educa- 
tion Service District project were focused on 
probability and statistics. The project served 20 
upper elementary and middle school teachers 
from the Hillsboro School District, including 
special education and English language learner 
teachers. The project partners were the school 
district, the Northwest Regional Education 
Service District, Pacific University, Linfield Col- 
lege, and the Northwest Regional Educational 
Laboratory. The project provided two weeks of 
professional development during the summer 
with follow-up activities during the school year 
(table A14). Teachers worked in school teams to 
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TABLE A12 

Greater Oregon Mathematics Partnership project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Core cadre: 56 hours plus 20 hours providing professional 
development 

Other participants: 20 hours 


Activity type 


Traditional — institutes, workshops 


Summer institute, workshops 




Reform — school- or classroom- 
based activities 




Collective 

participation 


Teachers from the same school or 
district work together 


No evidence provided in documentation 


Content focus 


Content knowledge 


Targeted focus on algebraic reasoning 
No evidence of content faculty involvement 




Teaching specific content 


No evidence identified 




How students learn content 


No evidence identified 


Active learning 


Observing or being observed 


Modeling of instructor during summer institute 




Planning for classroom 
implementation 






Reviewing student work 






Presenting leading discussions, 
writing 


Providing professional development (eight Core Cadre teachers) 


Coherence 


Connected to teachers' goals and 
other professional development 






Aligned with standards and 
assessments 


Project staff responsible for aligning activities with state 
standards, specific standards not identified 




Supports sustained communication 


Opportunities for ongoing communication for core cadre; not 
clear for other participants 


Partnership 

preconditions 


Existing relationships 


Proposal cited history of collaboration in the region; evidence of 
common projects conducted by the education service districts 




Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


Education service district served as lead organization 




Horizontal 






Sector 






Spatial 


Project served small rural schools in a large geographic area 


Interdependence 


Pooled 


Education service district was responsible for almost all aspects 
of the project 




Sequential 






Reciprocal 




Communication 




Not addressed 



develop lessons and they observed each other in 
the classroom. 

The Northwest Regional Education Service 
District project provided professional develop- 
ment that was sustained and that featured both 
traditional and reform activities. There was a tar- 
geted focus on content knowledge with multiple 



and varied opportunities for active learning. The 
project provided detailed information about the 
alignment of the professional development with 
standards. The documentation also addressed 
the collaborative process that the partners used 
to develop the project. The partnership had a 
horizontal structure with evidence of reciprocal 
interdependence. 
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TABLE A13 

High Desert project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 140 hours of professional development 


Activity type 


Traditional — institutes, workshops 


Summer institute, workshops 




Reform — school- or classroom- 
based activities 




Collective 

participation 


Teachers from the same school or 
district work together 


Teachers worked in district groups 

Project included activities to build participants' collaborative 
skills 


Content focus 


Content knowledge 


Multiple content areas: probability and statistics, geometry and 
measurement, algebra and number sense 
Content faculty member a co-principal investigator 




Teaching specific content 


Institute focused on standards and pedagogical content 
knowledge — how to teach specific content 




How students learn content 


No evidence identified 


Active learning 


Observing or being observed 


Teachers received classroom modeling and were observed by 
instructors 




Planning for classroom 
implementation 


Participants developed lesson plans 




Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 






Aligned with standards and 
assessments 


Activities designed to increase participants' understanding of 
National Council of Teachers of Mathematics standards; project 
activities analyzed to ensure alignment 




Supports sustained communication 


Participants encouraged to participate in ongoing conversations 
via online work 


Partnership 

preconditions 


Existing relationships 


Evidence of prior collaboration among four consortium districts; 
collaboration between schools and university not addressed 




Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


University was the lead organization 




Horizontal 






Sector 






Spatial 


The project served teachers in a large geographic area 


Interdependence 


Pooled 


University responsible for almost all aspects of the project 




Sequential 






Reciprocal 




Communication 




Not addressed 



Willamette Valley Watershed Partnership Project 

The Willamette Valley Watershed Partnership 
Project involved Oregon State University, Wil- 
lamette Education Service District, Western 



Oregon University, and an unspecified number 
of school districts. The science project focused on 
watershed ecology and engaged teachers in many 
field-based experiences. In year one the project 
identified 22 teachers to participate in the proj- 
ect, 1 at the elementary level, 15 middle school. 
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TABLE A14 

Northwest Regional Education Service District project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 80 hours of professional development 


Activity type 


Traditional — institutes, workshops 


Summer institute 




Reform — school- or classroom- 
based activities 


Lesson study 


Collective 

participation 


Teachers from the same school or 
district work together 


Activities focused in one district 
Teachers worked in school-based teams 


Content focus 


Content knowledge 


Targeted focus on probability and statistics; specific topics 
identified (for example, measures of central tendency and 
spread, ratios and proportions, sampling) 

Professional development provided by content faculty 




Teaching specific content 


No evidence identified 




How students learn content 


No evidence identified 


Active learning 


Observing or being observed 


Modeling of instructors, participants engaged in lesson study 




Planning for classroom 
implementation 


Participants engaged in lesson study 




Reviewing student work 


Participants engaged in lesson study 




Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 


Content selected based on teacher-identified needs 




Aligned with standards and 
assessments 


Alignment between project activities and state standards 
described in detail 




Supports sustained communication 


Ongoing communication supported via lesson study 


Partnership 

preconditions 


Existing relationships 


Partnership the result of two years of planning and collaboration 
among organizations 




Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 






Horizontal 


Partners shared leadership responsibilities 




Sector 






Spatial 




Interdependence 


Pooled 






Sequential 






Reciprocal 


Higher education partners, education service district, and school 
district worked together to design and deliver the professional 
development 


Communication 




Quarterly meetings 



and 6 high school. Of this group, 12 served as 
the comparison group and 10 participated in the 
professional development during year one. In ad- 
dition to a two -week summer institute, follow-up 
activities included three classroom visits from the 
faculty and three one-credit online courses during 



the school year (table A15). Teachers also observed 
and debriefed three demonstration lessons con- 
ducted by volunteer participants. 

The Willamette Valley Watershed Partnership Proj- 
ect provided sustained professional development 
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TABLE A15 

Willamette Valley Watershed Partnership Project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Over 80 hours of professional development 


Activity type 


Traditional — institutes, workshops 


Summer institute, online courses 




Reform — school- or classroom- 
based activities 


Classroom visits, demonstration lessons 


Collective 

participation 


Teachers from the same school or 
district work together 


Not designed to support collective participation 
There was collaboration among individual participants 


Content focus 


Content knowledge 


Targeted focus on watershed ecology; specific topics identified 
(for example, hydrology, landscape development) 

Professional development provided by content faculty 




Teaching specific content 


No evidence identified 




How students learn content 


Participants focused on misconceptions and how to promote 
conceptual change in students 


Active learning 


Observing or being observed 


Modeling of instructors, classroom visits, peer observation 




Planning for classroom 
implementation 


Developing and implementing units and lessons 




Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 






Aligned with standards and 
assessments 


Alignment between project activities and state standards 
described in detail 




Supports sustained communication 


Opportunities to engage in ongoing conversations online 


Partnership 

preconditions 


Existing relationships 


Evidence of prior relationships limited to the departments 
within the university 




Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


University served as the lead organization 




Horizontal 






Sector 






Spatial 


The project served teachers in a large geographic area 


Interdependence 


Pooled 


University responsible for designing and conducting 
professional development; education service districts handled 
administrative and financial tasks 




Sequential 






Reciprocal 




Communication 




Not addressed 



that featured both traditional and reform activities. 
There was a target focus on content grounded in 
exploring how to teach specific content. The align- 
ment between the project content and standards 
was described in detail. The partnership had a ver- 
tical structure with the university taking responsi- 
bility for most aspects of the project. 



Washington 

Washington funded four projects in year one. 

Two projects integrated mathematics and science, 
one project included both subject areas, and one 
project focused on mathematics. The partnerships 
served 258 teachers and were distributed across 
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the state— two were based in western Washington, contributed to the planning of project activities, 

one in central Washington, and one in the eastern Otherwise, the partners all had distinct roles. The 

region of the state. project was geographically complex, serving small 

and isolated rural school districts. 



The Washington MSP program required a two- 
week summer institute with follow-up. The state 
also tailored the requirements to emphasize Wash- 
ington priorities for mathematics and science, 
placing emphasis on pedagogical content knowl- 
edge and teacher collaboration. 

Eastern Washington Connections and Inquiry 

The Eastern Washington project integrated math- 
ematics and science around the theme of water 
quality. The project served 18 participants who 
taught at the middle and high school level at 10 
schools. The partners included Education Service 
District 101, eight to 10 school districts. Eastern 
Washington University, Community Colleges of 
Spokane, and Whitworth College. The project 
provided a four-day summer academy in year one, 
four days of workshops during the school year, and 
an additional day of flexible time that participants 
could use at their discretion (table A16). The con- 
tent of the follow-up workshops during the school 
year was tailored to the needs and requests of the 
participants. The pedagogical focus on the Eastern 
Washington project was inquiry, which was mod- 
eled for teachers through their participation in the 
summer academy. 

The professional development provided in the 
Eastern Washington project was sustained, with 
primarily traditional activities in addition to 
site -based activities that can be characterized as 
reform type. The project had targeted focus on 
content that integrated mathematics and science 
and provided multiple forms of active learning. 

The activities were designed to address the needs 
of the participating teachers. The project addressed 
mutual goals by including higher education faculty 
in the needs assessment interviews that shaped 
the project and by identifying partnership-related 
goals. The structure of the partnership was verti- 
cal, with evidence of reciprocal interdependence in 
that representatives from all partner organizations 



Partnership for Reform in Secondary 
Science and Mathematics 

The Partnership for Reform in Secondary Science 
and Mathematics partners included Washington 
State University-Vancouver, Education Service 
Districts 112 and 114, six school districts, and 
the Northwest Regional Educational Laboratory. 
The project served 45 middle school and high 
school teachers from 22 schools in six districts. 
Partnership for Reform in Secondary Science and 
Mathematics was designed around professional 
learning communities, with each group identify- 
ing the subject area for their professional develop- 
ment. The project provided a one-week summer 
institute and an academy day was held during the 
school year (table A17). Teachers were allocated 
time for monthly meetings of the professional 
learning communities and participated in peer 
observations. 

The Partnership for Reform in Secondary Sci- 
ence and Mathematics project provided sustained 
professional development that featured primarily 
reform activities in addition to a traditional sum- 
mer institute. The professional development did 
not focus on specific content — teachers worked 
in professional learning teams that determined 
individual areas of focus. The project provided 
multiple and varied forms of active learning, al- 
though the learning opportunities varied by team. 
The participants designed their work to meet their 
professional goals and the needs of their schools 
and districts. The partners pursued a mutual goal 
to develop new courses in science and mathematics 
for teachers. The structure of the partnership was 
vertical, but there was evidence that representatives 
from all partner organizations were involved in de- 
termining its design as well as additional evidence 
of reciprocal interdependence in the required coor- 
dination between district-based professional devel- 
opment activities and the projectwide activities. 



38 



ANALYSIS OF TITLE MB MATHEMATICS AND SCIENCE PARTNERSHIPS IN THE NORTHWEST REGION 



TABLE A16 

Eastern Washington project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 88 hours of professional development 


Activity type 


Traditional — institutes, workshops 


Academy, workshops 




Reform — school- or classroom- 
based activities 


School-based activities varied: observing, team planning 


Collective 

participation 


Teachers from the same school or 
district work together 


Teachers participated in school teams 

Opportunities for teachers from small districts to work together 


Content focus 


Content knowledge 


Targeted focus on water quality, specific topics identified (for 
example, micro-invertebrates, water quality testing, measuring 
stream flow) 

Professional development provided by content faculty 




Teaching specific content 


Project designed to integrate mathematics and science with 
pedagogical focus on inquiry 




How students learn content 


No evidence identified 


Active learning 


Observing or being observed 


Modeling of instructor, peer observation 




Planning for classroom 
implementation 


Planning units, designing assessment items 




Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 


Project designed to address need identified by the teachers 




Aligned with standards and 
assessments 


Statements that project was aligned with state standards and 
assessments, specific standards not identified 
Participants used standards in their work 




Supports sustained communication 


Opportunities to engage in collegial conversations via teacher 
teams 


Partnership 


Existing relationships 


Prior collaboration not identified 


preconditions 


Mutual goals 


Project included mutual goals 


Complexity 


Vertical 


Vertical structure: leadership shared by the education service 
district and university 




Horizontal 






Sector 






Spatial 


Geographically complex, serving isolated schools in a large area 


Interdependence 


Pooled 


In most aspects of the project, partners had independent 
responsibilities 




Sequential 






Reciprocal 


Planning project activities 


Communication 




Quarterly meetings 



Watershed Investigation Partnership 

The Watershed Investigation Partnership project 
served 25 middle school and high school teachers 
from 13 schools in 11 districts. The professional 



development integrated mathematics and science 
around the theme of stewardship of watersheds, 
with a pedagogical focus on inquiry. The partners 
included Quincy School District, the North Central 
Education Service District, and Central Washington 
University. Teachers participated in two summer 
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TABLE A17 

Partnership for Reform in Secondary Science and Mathematics project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 88 hours of professional development 


Activity type 


Traditional — institutes, workshops 


Institute 




Reform — school- or classroom- 
based activities 


Professional learning communities 


Collective 

participation 


Teachers from the same school or 
district work together 


Teachers participated in district-based teams 


Content focus 


Content knowledge 


No project-level content focus 




Teaching specific content 


Most of the teams focused professional inquiries on how to 
teach content 




How students learn content 


Evidence not identified 


Active learning 


Observing or being observed 


Some teams engaged in peer observation 




Planning for classroom 
implementation 


Some teams developed lessons and engaged in collaborative 
planning 




Reviewing student work 


All participants analyzed student work 




Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 


Teams designed and selected their own work based on their 
needs 




Aligned with standards and 
assessments 


One focus for project work was aligning instruction and 
materials to state standards 




Supports sustained communication 


Opportunities to engage in collegial conversations via teacher 
teams 


Partnership 


Existing relationships 


Project based on ongoing relationships and prior collaboration 


preconditions 


Mutual goals 


Mutual goal to develop new courses for teachers 


Complexity 


Vertical 


Project lead by university 




Horizontal 






Sector 






Spatial 


The project served teachers in a large geographic area 


Interdependence 


Pooled 






Sequential 






Reciprocal 


Partners involved in designing the project, coordinating district- 
based professional development with project activities 


Communication 




Leadership meetings: three per year; steering committee: 
monthly 



institutes and during the school year, the project 
provided two days of on-site support and conducted 
distance learning activities (table A 18 ). The pedagog- 
ical focus was inquiry— both engaging teachers in 
inquiry and how to apply inquiry-based education. 

The professional development provided in the 
Watershed Investigation Partnership project was 
sustained, with primarily traditional activities 



in addition to on-site support. The project had an 
integrated content focus based on a watershed 
stewardship theme with multiple and varied op- 
portunities for active learning. The structure of the 
partnership was vertical— the university and the 
lead school district took responsibility for the bulk 
of the project activities. The project was geographi- 
cally complex, serving small and isolated rural 
school districts in a large area of the state. 
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TABLE A18 

Watershed Investigation Partnership project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Approximately 96 hours of professional development 


Activity type 


Traditional — institutes, workshops 


Institutes 




Reform — school- or classroom- 
based activities 


On-site support 


Collective 

participation 


Teachers from the same school or 
district work together 


Teachers participated in school or district teams 
Some teachers from small, rural schools participated in cross- 
district teams 


Content focus 


Content knowledge 


Theme: "Stewardship of the Environment and Watershed 
Investigation" 

Professional development provided by content faculty 
Designed to integrate mathematics and science 




Teaching specific content 


Pedagogical focus on inquiry 




How students learn content 


Evidence not identified 


Active learning 


Observing or being observed 


Modeling of instructor 




Planning for classroom 
implementation 


Developing projects, planning for assessment 




Reviewing student work 






Presenting leading discussions, 
writing 


Reporting on the results of classroom projects 


Coherence 


Connected to teachers' goals and 
other professional development 


Participants conducted a needs assessment to plan their work 




Aligned with standards and 
assessments 


Participants used state standards and assessments to guide their 
work 




Supports sustained communication 


Opportunities to engage in collegial conversations via teacher 
teams 


Partnership 


Existing relationships 


No evidence of prior collaboration 


preconditions 


Mutual goals 


All goals teacher- or student-focused 


Complexity 


Vertical 


Leadership shared by university and lead district 




Horizontal 






Sector 






Spatial 


Geographically complex, serving isolated schools in a large area 


Interdependence 


Pooled 


University responsible for most aspects of the project 




Sequential 






Reciprocal 




Communication 




Not addressed 



Whatcom-Skagit 

The project provided professional development in 
mathematics for 174 K-14 and preservice teach- 
ers. The partners included Western Washington 
University, 12 school districts, Bellingham Techni- 
cal College, Skagit Valley College, and Whatcom 
Community College. The project conducted a 



one-week summer institute and provided monthly 
seminars during the school year in which the 
participants varied from activity to activity 
(table A19). The project was designed to represent 
the breadth of mathematics content rather than 
the depth of a specific content strand. There were 
also multiple pedagogical practices covered in the 
professional development. 
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TABLE A1 9 

Whatcom and Skagit project features 



1 Criteria 


Subcriteria 


Project features I 


Duration 




Documentation does not provide information about how many 
hours of professional development participants received 


Activity type 


Traditional — institutes, workshops 


Institute, seminars 




Reform — school- or classroom- 
based activities 




Collective 


Teachers from the same school or 


Not designed to support collective participation 


participation 


district work together 




Content focus 


Content knowledge 


Professional development covered multiple content areas in 
mathematics 

Content faculty were identified in the proposal 




Teaching specific content 


Professional development covered multiple pedagogical 
strategies 




How students learn content 


Evidence not identified 


Active learning 


Observing or being observed 


Modeling of workshop instructors 




Planning for classroom 
implementation 






Reviewing student work 






Presenting leading discussions, 
writing 




Coherence 


Connected to teachers' goals and 
other professional development 


Participants selected activities to participate in 




Aligned with standards and 
assessments 


Statements that project was aligned with a variety of standards 
(for example. National Council of Teachers of Mathematics, 
American Association for the Advancement of Science, essential 
academic learning requirements), specific standards not 
identified 




Supports sustained communication 


Project goal to develop a network of teachers maintained via 
e-mail 


Partnership 


Existing relationships 


Based on prior collaboration 


preconditions 


Mutual goals 


Mutual goal: improving teacher education program 


Complexity 


Vertical 


University responsible for almost all aspects of the project 




Horizontal 






Sector 






Spatial 


The project served teachers in a large geographic area 


Interdependence 


Pooled 


In most aspects of the project, partners had independent 
responsibilities 




Sequential 






Reciprocal 


Planning activities 


Communication 




Monthly meetings 



The Whatcom-Skagit project provided a series 
of workshops that addressed a variety of con- 
tent areas in mathematics. Because participants 
varied from activity to activity, it is not possible to 
determine the duration of this project. According 
to the documentation, active learning opportu- 
nities were limited to observing the instruction 



modeled during the workshops. The partners pur- 
sued a mutual goal: one of the project objectives 
for year two was improving the teacher prepara- 
tion program. The documentation indicated that 
the partnership had a vertical structure, with the 
university responsible for almost all aspects of the 
project. 
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APPENDIX B 

ANALYSIS OF THE TITLE MB MSP PROFESSIONAL 
DEVELOPMENT EVALUATION PROJECTS BY STATE 

This appendix addresses the second research ques- 
tion, which asks about the nature of the evaluation 
of the Title IIB MSP projects in the Northwest 
Region. The project proposals, year one annual 
reports, and interview data were analyzed through 
the evaluation framework to examine the extent to 
which the Title IIB MSP evaluations addressed the 
criteria for quality in evaluation articulated in the 
Joint Committee on Program Evaluation Stan- 
dards (JCSEE, 1994). 



Alaska 

Journeys in Mathematics 

The purpose of the evaluation is not identified in 
either the proposal or the year one annual report. 
The evaluator is internal to the project and is one 
of the university faculty implementing the profes- 
sional development. Evaluation questions were not 
specifically identified, nor was the methodological 
approach specified. Information sources included 
only the teachers and records of professional 
development activities. Data sources included the 
use of an efficacy instrument, the mathematics 
assessment developed by the Study of Instruc- 
tional Improvement, and the Diagnostic Teacher 
Assessment for Mathematics and Science, number 
and computation test. Data collection procedures 
and schedule were not specified. Neither quantita- 
tive nor qualitative data analysis procedures were 
reported. 



Idaho 

All of the Idaho projects had relatively weak 
evaluation designs that provided minimal infor- 
mation about the nature and effect of the profes- 
sional development implemented. These designs 
did not provide a clear alignment among project 
goals or objectives, evaluation questions, and data 
sources. Procedures and schedules of information 
collection were not well described. Information 



sources were not well identified, and data analysis 
procedures were not articulated. In the interview 
with the state education agency staff responsible 
for the administration of the Title IIB MSP pro- 
gram, lack of capacity in evaluation was identified 
as the crucial issue. In addition, this individual 
argued that the Title IIB MSP projects as imple- 
mented by the state education agency are not the 
proper venue for rigorous research on professional 
development. 

Classroom Assessment for Mathematics Performance 

The purpose of the evaluation was identified in 
both the proposal and the year one annual report 
as being both formative and summative in nature. 
An external evaluator was identified, but appar- 
ently not used. The evaluation questions were not 
specifically identified, and the number of objec- 
tives for the project were numerous. The method- 
ological approach was not specified. Information 
sources, including teachers and students were 
vaguely described, and instruments included 
surveys of technology use, instructional practices, 
focus group interviews, and case studies. Data col- 
lection procedures and schedule were not clearly 
articulated, and neither quantitative nor qualita- 
tive data analysis procedures were reported. 

Coeur d'Alene — Mathematics 

The purpose of the evaluation was not specified 
in either of the documents as being either forma- 
tive or summative in nature. Evaluation questions 
were articulated and focused on the impact of 
project activities on teachers’ knowledge, beliefs, 
and practices; on students; and on school culture. 
An external evaluator was identified but not used, 
as project staff conducted the evaluation. The 
methodological approach was not specified. Infor- 
mation sources and sampling strategies were not 
clearly identified. Instruments included state test 
scores and nonspecific project developed surveys. 
Data collection procedures and schedule were 
not clearly specified, and neither quantitative nor 
qualitative data analysis procedures were reported 
in the evaluation report. 
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Developing Mathematical Thinking 

No project-specific evaluation plan was communi- 
cated in either the proposal or the year one annual 
report. In an interview Idaho state agency staff 
indicated that the evaluation plan was included in 
a separate proposal Five project goals were identi- 
fied instead of evaluation questions. The evaluator 
is not specified and appears to be the principal 
investigator. The methodological approach was not 
specified. Information sources included only the 
teachers. Instruments included a project-designed 
knowledge and pedagogy inventory, and observa- 
tions of teaching were conducted. Data collection 
procedures and schedule were not clearly speci- 
fied, and neither quantitative nor qualitative data 
analysis procedures were reported in the evalua- 
tion report. 

Twin Falls — Curriculum Improvement and 
Alignment in Middle School Mathematics 

Purposes of evaluation were identified as both 
formative and summative in both the proposal 
and year one annual report. No evaluator was 
acknowledged, and the annual report was writ- 
ten by the school district personnel responsible 
for the coordination of the teachers in the project. 
The evaluation question appeared to focus on 
one objective, with additional mention of im- 
proving mathematics instruction and increasing 
student mathematics skills. The methodological 
approach was not specified, nor were the informa- 
tion sources and sampling specified. Instruments 
included state test scores, participant surveys, and 
review of project documents. The state test for 
middle school children was used as the only mea- 
sure of increases in teacher content knowledge. 
Data collection procedures and schedule were 
not clearly specified, and neither quantitative nor 
qualitative data analysis procedures were reported 
in the evaluation report. 



Montana 

The evaluation designs of five of the six Montana 
projects were relatively weak and unspecific. Only 



one of the projects attempted a comparison group 
design and only at the student level. The other five 
projects articulated evaluation designs that did not 
provide a clear alignment among project goals or 
objectives, evaluation questions, and data sources. 
Procedures and schedules of information collec- 
tion were not well described. Information sources 
were not well identified, and data analysis pro- 
cedures were not articulated. The interview with 
the Montana Office of Public Instruction identi- 
fied evaluation capacity as a key weakness of the 
partnerships. 

Billings 

The purposes of the evaluation were not ar- 
ticulated. The evaluation questions were a direct 
restatement of the project goals and were not well 
connected to data sources. The evaluator was 
external to the project, though credentials were 
not provided. The methodological approach was 
not specified, though there was some indication 
of the use of a comparison group. However, that 
comparison group was not identified and no data 
sources were put forward for additional teachers. 
Information sources included the teachers and 
their students. Instruments included a PRAXIS 
mathematics content test for teachers, tracking of 
number of highly qualified teachers, student Mon- 
tana state assessment, and standardized scores. 
Data collection procedures and schedule were 
not clearly specified, and neither quantitative nor 
qualitative data analysis procedures were reported 
in the evaluation report. 

Creating Opportunities in Mathematics 

for Exemplary Teaching 

The evaluation purposes were not clearly identi- 
fied in terms of formative or summative uses of 
the report. The evaluation questions were a direct 
restatement of project goals and were neither well 
specified nor connected to data sources. The evalu- 
ator was internal to the project, and credentials 
for evaluation were not clearly communicated. 

The methodological approach was not specified. 
Information sources were identified as impacts 
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on both teachers and students. Instruments 
included a teacher content instrument and survey 
designed by the project, teacher self-reports, and 
project documentation. Student achievement was 
measured with the Montana state assessment. 

Data collection procedures and schedule were not 
clearly described, and neither quantitative nor 
qualitative data analysis procedures were reported 
in the evaluation report. 

Flathead and Salish Kootenai College 

Evaluation purposes were communicated as being 
both formative and summative. The evaluation 
questions were clearly articulated and related to 
project goals and objectives and to data sources. 
The evaluator was external to the project, and 
credentials were clearly communicated. A com- 
parison group evaluation design was identified in 
the proposal, but only comparison student data 
were used. Information sources identified included 
treatment group teachers and both treatment 
and comparison group students. Instruments 
used included the Diagnostic Teacher Assessment 
in Mathematics algebra test, a project-designed 
test of teacher algebra content knowledge, and 
a technology survey. Student data included post 
test of student mathematics content knowledge 
using a project-designed instrument, mathematics 
self-efficacy survey, and Montana state assessment 
data. Data collection procedures and schedule 
were clearly specified. Descriptive and compara- 
tive statistics were described, and qualitative data 
use was not reported. 

High School Chemistry in Western Montana 

The purposes of the evaluation in terms of forma- 
tive or summative use of information were not 
specified. The evaluation questions were the direct 
restatement of project goals and not well con- 
nected to data sources. The evaluator was internal 
to the project. The methodological approach was 
not specified. Information sources were project 
teachers and a subset of students. Instruments 
used to collect data about increases in chemistry 
content knowledge and instructional practices for 



teachers included the American Chemical Society 
content test for students, which indicated a ceiling 
effect, and the Learning Cycle Test. No student 
data were identified. Data collection procedures 
and schedule were not clearly specified, and nei- 
ther quantitative nor qualitative data analysis pro- 
cedures were described in the evaluation report. 

Improving Science Teaching 

The purpose of the evaluation was not specified. 
The evaluation questions were the project goals 
and not well connected to data sources. The evalu- 
ator is internal to the project. The methodological 
approach was not specified. Information sources 
were not well identified, but included both teach- 
ers and students. The instruments were not well 
characterized, and data sources included teacher 
lesson plans, classroom observations, and a modi- 
fication of a teacher test from an Oregon Title IIB 
MSP watershed ecology project. Student-level data 
included a project-designed subset of test designed 
for an Oregon MSP project that included questions 
for students and an inquiry-based science attitude 
survey. Data collection procedures and schedule 
were not clearly specified though a comparison 
group of students is indicated. The comparison 
group is identified as a convenience sample with- 
out indication of the group with whom the group 
is to be compared. Some quantitative and qualita- 
tive data analysis procedures were reported in the 
evaluation report. 

Montana Science and Mathematics Consortium 

Evaluation purposes were characterized as forma- 
tive, as continuous improvement of the project was 
identified as the key use of evaluation information. 
The evaluation questions are a subset of multiple 
project goals. The evaluator is not specified. The 
methodological approach was not specified. The 
information sources were not clearly identified. 
Instruments were not identified. Data collection 
procedures and schedule were not clearly speci- 
fied, and neither quantitative nor qualitative data 
analysis procedures were reported in the evalua- 
tion report. 
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Oregon 

The Oregon program included additional points 
for an experimental or quasi-experimental evalua- 
tion design. All four projects attempted a compari- 
son design with varying levels of success. Small 
sample sizes were evident, and the description of 
sampling strategies was weak. Projects identified 
specific instruments or designed instruments to 
test teacher and student content knowledge, and 
the schedule of data collection was articulated. 

The Survey of Enacted Curriculum was used in 
two of the four projects. Quantitative data analysis 
procedures were more consistently identified than 
qualitative data analysis procedures. 

Greater Oregon Mathematics Partnership 

The purposes of the evaluation were not clearly 
identified. Evaluation questions were restatements 
of the project goals and were not aligned with data 
sources. The evaluator is co -principal investigator 
on the project. A comparison group evaluation 
design was included in the proposal, but there 
was no evidence of such in the year one report. 
Information sources included in the proposal were 
the project teachers and students. Instruments 
included a COMPASS pre-college mathemat- 
ics assessment and project-specific instrument 
for teachers and the Oregon state assessment for 
students. Only minimal data were reported in the 
year one report. Data collection procedures and 
schedules were not specified. Quantitative data 
analysis procedures were not well described and 
the only qualitative data analysis was the identifi- 
cation of illustrative quotes. 

High Desert 

The evaluation purposes were identified as being 
both formative and summative. The evaluation 
questions were clearly articulated and related to 
project goals and objectives and to data sources. 
The evaluator was external to the project staff and 
credentials were included. An experimental design 
was included in the proposal, the school was 
the unit of random assignment, and the sample 



size was very small. The sampling strategy was 
conducted by project staff, and data collection pro- 
cesses were well identified. Specific instruments 
for teacher data included the Diagnostic Teacher 
Assessment in Mathematics (all four tests), 
PRAXIS, and a project-specific professional devel- 
opment survey. Student outcome data included the 
Oregon state mathematics assessment for students 
reported at the individual student level. Descrip- 
tive and comparison statistics were described to 
analyze quantitative data. No qualitative data 
analysis was reported. 

Northwest Regional Education Service District 

The evaluation purposes were identified as being 
both formative and summative. The evaluation 
questions were clearly articulated and related to 
project goals and objectives and to data sources. 
The evaluator was external to the project staff and 
credentials were included. An experimental design 
was implemented, the school was the unit of ran- 
dom assignment, which was conducted by project 
staff, and the sample size was very small. The 
sampling strategy was not adequately described. 
Specific instruments for teacher data included 
the Diagnostic Teacher Assessment in Mathemat- 
ics (probability and statistics test), the Survey of 
Enacted Curriculum (instructional beliefs and 
practices survey), document analysis of lesson 
study reports, and the use of the Reformed Teach- 
ing Observation Protocol. Student outcome data 
included the Oregon state mathematics assessment 
for students reported at the classroom level. De- 
scriptive and comparison statistics were described 
to analyze quantitative data. No qualitative data 
analysis was reported. 

Willamette Valley Watershed Partnership Project 

The evaluation purposes were identified as being 
both formative and summative. The evaluation 
questions were clearly articulated and related to 
project goals and objectives and to data sources. 
The evaluator was external to the project staff and 
strong credentials were included. A comparison 
group evaluation design was implemented, the 
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individual teacher was the unit of assignment, 
and the sample size was small. Teachers were al- 
lowed to self-select into treatment and comparison 
groups. Information sources included both project 
teachers and their students, though only students 
at the middle school level were administered 
the science content test. Specific instruments for 
teacher data included a project-designed test for 
teacher content knowledge, the Survey of Enacted 
Curriculum (instructional beliefs and practices), 
and documentation of inquiry science lesson 
plans. Student data included Oregon state science 
assessment for those students in grades five and 
eight and test of middle school student knowledge 
of watershed ecology, hydrology, and geology 
concepts designed and analyzed by the Northwest 
Evaluation Association. Descriptive and compari- 
son statistics were described to analyze quantita- 
tive data; however, qualitative data analysis was 
not well specified. 



Washington 

Across the four projects in Washington evalua- 
tion design and implementation was weak. Several 
projects did not have well articulated goals and 
outcomes that demonstrated the impact of the pro- 
fessional development experienced by the teachers. 
Methodological approaches to the evaluations 
were not well characterized. Instrumentation 
included project-designed surveys of participant 
reactions, and the reporting of student data was 
at the school level. Data collection procedures and 
schedules were vaguely described, and data analy- 
sis was not well characterized. 

Eastern Washington Connections and Inquiry 

Evaluation purposes were not clearly identified in 
either the proposal or the year one annual report. 
Evaluation questions were clearly identified and 
tied to project goals. The evaluator was external 
to the project, and credentials were identified. The 
methodological approach for the evaluation was 
not specified. Information sources are not clearly 
identified, but they included project teachers and 
students at the school level. Instruments included 



a project-designed teacher survey, the Survey of 
Enacted Curriculum, a project-designed classroom 
observation protocol, student classroom assess- 
ments, an affective survey, and state assessment 
in data mathematics and science. Data collec- 
tion procedures and schedule were not specified. 
Quantitative analysis procedures included descrip- 
tive statistics, but qualitative analysis was not well 
characterized. 

Partnership for Reform in Secondary 
Science and Mathematics 

The purpose of the evaluation was identified as 
formative. Evaluation questions were specified and 
based on objectives and benchmarks. The evalu- 
ator was external, and credentials were specified. 
The methodological approach was not specified, 
and information sources were not clearly identi- 
fied. Instruments included a project-designed 
survey, use of Professional Learning Community 
survey (teacher self-report), focus group inter- 
views, and student state assessment data reported 
by school. The Survey of Enacted Curriculum 
instructional beliefs and practices instrument was 
administered at the start of the project. Data col- 
lection procedures were identified. Comparative 
statistics on surveys were reported, and qualitative 
data analysis was not well characterized. 

Watershed Investigation Partnership 

The evaluation purpose was identified as being 
both formative and summative in nature. Evalu- 
ation questions were identified and aligned with 
the project objectives. The evaluator appeared 
to be internal to the project, but the relationship 
was not well characterized. The methodologi- 
cal approach was not well specified. Information 
sources were project teachers, document review of 
project activities, and student data. Instruments 
included an unknown teacher interview, surveys, 
and observations. Student data included teacher- 
developed content tests that were not described. 
The data collection procedures and schedule were 
not well specified. The analysis procedures were 
not described. 
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Whatcom and Skagit 

The evaluation purpose was identified as being for- 
mative for the first year of the project. Evaluation 
questions were based on National Research Coun- 
cil principles for human learning, not project goals 
and objectives. The evaluator was internal from 
one of the participating schools. The methodologi- 
cal approach was not well specified. Information 



sources included teachers and students. Instru- 
ments included teacher reaction and perceptions 
to the project activities and workshops, interviews, 
and observations. Student information was the 
school-level state assessment in mathematics. 

Data collection procedures and schedule were not 
identified. Quantitative analysis was descriptive 
statistics of teacher reactions and perceptions, but 
qualitative data analysis was not conducted. 
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APPENDIX C 
INTERVIEW PROTOCOL 



A. What does the professional development 
look like implemented? 

1. Describe the design of your professional de- 
velopment activity. (If I were a teacher in your 
project, what would I experience? Did you use 
a pre-set curriculum? ) 

Probe: What variations have you had to make 
from your original design (that is, in the grant)? 

2. What are the underlying theories about how 
people learn that support the professional 
development activity? 

Probe: Walk me through an example that 
would help me see these principles in action. 

3. How is the math or science content evident in 
project activities? 

4. How does content relate to pedagogy and 
strategies? 

Probe: Provide an example of what this rela- 
tionship might look like in the professional 
development setting. 

5. To what extent are collaborative networks 
of teachers evident in the project design and 
implementation? 

6. How does your design call for the active involve- 
ment of your school partners? administrators? 



B. How does your project relate to what is considered 
effective professional development? 

7. Duration: For how much time are teachers 
directly engaged in professional development? 

Probe: Do you feel that is enough time to 
develop deep understanding and the skills to 
implement what was taught? 



8. Focused on teacher work: How is the profes- 
sional development activity directly linked to 
required school curriculum content and state 
and district standards? 

9. How students learn: What provision is there 
for examining how students learn and 
how to effectively gauge or measure that 
learning? 

10. New learning curriculum: How does the 
professional development activity prepare 
teachers to use the new content they are 
learning? 



C. How do you know that your professional 
development activity is effective? 

11. How is the evaluation designed to identify 
project outcomes and promising practices? 

12. How do you know that the teachers learned 
the science or math content that was the focus 
of the professional development? 

13. How do you know that the teachers have used 
the content that was the focus of the profes- 
sional development? 

14. How do you know that the teachers used 
pedagogical skills that matched what you 
presented? 

Probe: For instance, if your project focused on 
eliciting student misconceptions, do you have 
evidence of how the teachers tried that in the 
classroom? 

15. What evidence have you collected that teach- 
ers implemented newly learned curricular 
activities? 

Probe: Can you give me an example of a math 
(or science) activity that you taught teachers 
during the summer and how it was imple- 
mented in a teacher s classroom? 
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D. What are the specifics of the evaluation design? 

16. How did you determine how to measure the 
outcomes of the professional development 
activity? 

17. Does your evaluation design use comparison 
groups of teachers? (If not, then why not?) 

Probe: Why did you choose to use a compari- 
son group design? 

Probe: How were teachers assigned? 

Probe: If quasi-experimental, how have the 
comparison groups been identified (what 
characteristics?) 

Probe: How much attrition has there been? 

18. Describe how your evaluation measures the 
changes in teacher science or mathematics 
content knowledge. 

19. Describe how your evaluation measures the 
improvement in teaching or pedagogical skills. 



E. How are you connecting the professional 
development activity to student learning 
(state tests and other measures)? 

20. Describe how your evaluation measures proj- 
ect impact on student learning. 

21. Describe how you have used state assessment 
data to measure change in student achieve- 
ment resulting from project activities. 

22. Describe how you link professional develop- 
ment, teacher knowledge, and instructional 
strategies with student achievement. 

F. Involvement of the partners 

23. How has the project strengthened the involve- 
ment of higher education in the professional 
development of teachers? 



24. How were the content faculty involved in the 
design of the project activities? 

Interview protocol — state education agency staff 

1. Here is what we re doing — the question we 
will return to in the end of this conversation 
is ... is this useful, and how? 

2. What are the aspects of your work in directing 
the state projects? 

3. You indicated that the request for proposals 
for these projects came from a template. Do 
you know where the template came from? 

4. What do the projects in your state look like 
now? 

5. Are you getting projects you want? 

6. In our initial interview, you indicated that 
both teachers reaching highly qualified 
teacher status and adequate yearly progress 
were policy issues. Are those still the main 
policy focus? 

7. Are there other policy issues that are emerg- 
ing in Alaska on which the Title IIB MSP 
program has an influence? 

8. Are there other programs that have an influ- 
ence on highly qualified teacher and adequate 
yearly progress issues? If so, what is the 
relationship of these with the Title IIB MSP 
program? 

Does Title IIA still go to districts and 
then flow through to schools? 

How does Title IIB connect to Title 
IIA in terms of state policy? 

9. What is it that this project can do to address 
policy issues and evaluation issues? 
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APPENDIX D 
METHODOLOGY 



Data sources 

The sources of data for this research include the 
text from the elementary and secondary education 
legislation that addressed Title IIB Mathematics 
and Science Partnerships (www.ed.gov/policy/ 
elsec/leg/esea02/pg26.html). The year one requests 
for proposals initiated by the five states were col- 
lected. Each state education agency provided cop- 
ies of the funded proposals and year one annual 
reports. Interviews were conducted with project 
principal investigators and evaluators at the end 
of the first year of the projects during the fall of 
2005 as part of the scope of work of the Northwest 
Eisenhower Regional Consortium. A series of 
interviews were conducted with state education 
agency staff responsible for the implementing 
the Title IIB program in the five states. Table DI 
indicates the total number of data sources used in 
this analysis. Interview protocols are included in 
appendix D. 

Analysis framework — professional development 

The nature of the professional development being 
implemented in each of the projects was exam- 
ined using an analytic framework based on the 
National Evaluation of the Eisenhower Profes- 
sional Development Program (Desimone et al., 
2002; Caret, Birman et al, 1999; Caret, Porter 
et al, 2001; Porter et al, 2000). The framework 
is organized around six features of high quality 



professional development that were identified in 
that evaluation of mathematics and science pro- 
grams: duration, activity type, collective participa- 
tion, content focus, active learning, and coherence 
(table D2). 

The Eisenhower framework is one of many pos- 
sible lenses with which to analyze and describe 
professional development. A range of alternative 
frameworks were considered for use in the de- 
scriptive analysis (for example, American Eedera- 
tion of Teachers, 2002; Loucks-Horsley et al., 1998; 
National Staff Development Council, 2001). The 
Eisenhower framework was selected because it is 
grounded in existing research and was tentatively 
validated with self-report data from teachers, it 
is widely known in the field, and it is specifically 
related to the content areas of mathematics and 
science. The Eisenhower criteria are reflected in 
the definition of professional development put 
forth in the No Child Left Behind Act, which 
provides guidelines for designing projects such as 
the Mathematics and Science Partnerships. Some 
of the parameters of the definition include a focus 
on teachers’ knowledge of academic subjects, skills 
to help students meet challenging standards, and 
understanding of effective instructional strategies 
that are grounded in scientifically based research. 
The definition establishes that professional devel- 
opment must be connected to school and district 
improvement plans and aligned with standards, 
curricula, and assessments. Another emphasis 
is on activities that are sustained, intensive, and 
classroom-focused rather than short-term work- 
shops or conferences. 



TABLE DI 

Data sources 



State 


Year one request 
for proposal? 


Funded proposals 


Year 1 annual reports 


Interviews 


Alaska 


Yes 


1 


1 


3 


Idaho 


Yes 


4 


4 


9 


Montana 


Yes 


6 


6 


14 


Oregon 


Yes 


4 


4 


13 


Washington 


Yes 


4 


4 


10 
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TABLE D2 

Analysis framework — professional development 



Structural features 


Key feature and definition 


Subcriterion 


Program duration and frequency 

The number of hours of professional development provided 
by the project and the timespan are adequate to enable 
teachers to learn new ideas and incorporate them into their 
practice. 


Activity type 

Traditional activities are more likely to take place outside 
of the school, while reform activities are more likely to be 
integrated into teachers' work. 


Traditional activities 

• Within-district workshops or conferences. 

• Coursesfor college credit. 

• Out-of-district workshops or conferences. 

Reform activities 

• Teacher study groups. 

• Teacher collaboratives, networks, or committees. 

• Mentoring. 

• Internships. 

• Resource centers. 


Collective participation among teachers 

The project provides opportunities for participants to work 
with other teachers from the same school or district. 


Core features 


Content focus 

The professional development is grounded in subject matter 
and addresses how to teach specific content to students. 


• Emphasis on content knowledge. 

• Emphasis on how students learn specific content. 

• Emphasis on methods of teaching specific content. 


Active learning 

The project provides opportunities for teachers to actively use 
what they are learning. 


• Observing or being observed. 

• Planning for classroom implementation. 

• Reviewing student work. 

• Conducting presentations or demonstrations, leading 
discussions, writing reports or plans. 


Coherence 

The project activities are connected to other professional 
development, align with standards, and support ongoing 
communication. 


• Connected to other professional development and 
teachers' goals. 

• Aligned with standards and assessments. 

• Supports teachers in developing sustained, ongoing 
communication. 



Source; Adapted from the Professional Development Program Quality Rubric developed by the Council of Chief State School Officers. 



The Eisenhower criteria are not based on conclu- 
sive evidence that the six identified features of 
professional development cause improvements in 
teacher knowledge or practice. In general, the field 
of professional development lacks such evidence 
because evaluations have typically focused on par- 
ticipants’ satisfaction with their experiences and 
self-reports of impact. Programs such as the Math- 
ematics and Science Partnerships are intended to 
begin filling this gap in the knowledge of effective 
professional development. 



Development of the framework 

The Council of Chief State School Officers (CCSSO) 
developed a rubric based on the Eisenhower 
criteria for their evaluation of MSP-supported 
professional development, Improving Evaluation of 
Professional Development in Mathematics and Sci- 
ence Education (Blank, 2006). In the CCSSO study 
the rubric was used to provide a rating for the 27 
National Science Eoundation and Title IIB MSP 
projects in 15 states included in the study based on 
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an expert panel analysis and moderation process. 
This rating was a quantitative judgment based on 
document analysis and repeated interviews with 
project principal investigators and evaluators. The 
rubric served as an initial model for the analysis 
framework in this Regional Educational Labora- 
tory Fast Response Task, but the intended use of 
the framework was not to give a rating to each proj- 
ect because the analysis had to be made on existing 
documents and a single set of interviews with proj- 
ect staff that took place before the analysis began. 
Instead, the framework for this study was used to 
develop a profile of each project that describes the 
attributes of the professional development. 

In the first stages of the Northwest Regional 
Educational Laboratory study it was initially 
planned that the CCSSO rubric would be used to 
analyze the professional development and develop 
the project descriptions. Closer examination of 
the rubric revealed that the original Eisenhower 
evaluation categories were a better fit to the Title 
IIB MSP projects in the Northwest Region given 
the intentional modeling of the programs on 
previous Eisenhower professional development. 

For example, the activity type feature was omitted 
from the CCSSO rubric. Collective participation 
was expanded to include teachers from the same 
grade level and content area, rather than focus- 
ing on teachers from the same school or district. 
CCSSO also added subcriteria to active learn- 
ing and coherence that were not included in the 
original research. In addition. Northwest Regional 
Educational Laboratory staff did not have access 
to the rationale for the modifications that CCSSO 
made to the original Eisenhower rubric. This is 
not meant as a critique of the CCSSO process. 
Rather than modeling the framework directly on 
the CCSSO rubric, the analysis of the Northwest 
Region projects is based more closely on the defi- 
nitions of each feature of professional development 
from the original research. Appendix E provides 
a comparison of the criteria from CCSSO with the 
analysis framework used in this report. 

The analysis of the professional development also 
includes criteria that examine the nature of the 



partnerships that are demonstrated in project docu- 
ments and through interviews. It has been more 
problematic to structure the criteria for what consti- 
tutes an effective partnership. The work of Gordon 
Kingsley and his group at the Georgia Institute of 
Technology has informed the development of the 
characteristics to describe “partnership” in the Title 
IIB MSP projects (table D3). A partnership, for the 
purpose of this analysis, is a group of entities (orga- 
nizations such as schools, colleges or universities, 
and for-profit or non-profit companies) that work 
together to accomplish a set of mutual goals. 

Key traits of successful partnerships include 
measures of supporting preconditions, complexity, 
interdependence, and communication (Kingsley & 
O’Neil, 2004; Kingsley & Waschak, 2005). Partner- 
ship preconditions include existing relationships 
between organizations prior to the development 
of the partnership. An additional precondition is 
the extent to which the needs of the partners are 
congruent and the partnership enables them to 
pursue mutual goals. 

Complexity encompasses several dimensions to 
describe the structure of a partnership. A partner- 
ship with a vertical structure is hierarchical, and 
a partnership with a horizontal structure includes 
peer organizations on the same level. Partnerships 
with sector complexity include organizations with 
different areas of work. Spatial complexity refers 
to size of the geographic area that the partnership 
serves. 

Interdependence characterizes the extent to which 
partners depend on each other for resources or 
materials and how they accomplish their work. 
Partners with reciprocal interdependence share 
their work back and forth. With sequential in- 
terdependence, the work of one partner will feed 
into the work of another. Pooled interdependence 
characterizes partnerships in which the members 
work independently for the most part. 

The final partnership dimension is communication, 
which refers to the frequency of interaction among 
partners. The partnership characteristics of the 
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TABLE D3 

Analysis framework — partnership 


1 Key feature and definition 


Subcriteria 1 


Preconditions 

The extent to which the partnership is based on existing 
collaboration and mutual goals shared among the partners. 


• Embeddedness (relationships between organizations prior to 
the development of the partnership). 

• Strategic needs (congruence of complementarity in the 
needs of the partners, mutual goals). 


Complexity 

The number of different organizations and activities 
involved in the partnership. 


• Vertical (hierarchy, clear lead organization with clear 
followers). 

• Horizontal (number of peer organizations on the same level 
with the same tasks). 

• Sector (number of partners from private, public, not-for- 
profit). 

• Spatial (number of different geographical locations). 


Interdependence 

The extent to which partners depend on each other for 
resources or materials. 


• Pooled (each works fairly independently). 

• Sequential (the work of one feeds into the work of another). 

• Reciprocal (each partner must share work). 


Communication 

The frequency of interaction. 



Source: Adapted from the Professional Development Program Quality Rubric developed by the Council of Chief State School Officers. 



projects in this analysis were more difficult to iden- 
tify, and the inferences made are more tentative. 

Data analysis 

In conducting the descriptive analysis of the 
professional development, the proposals, year one 
evaluation reports, and interviews were examined 
to identify evidence related to each of the criteria 
and subcriteria in the analysis framework. The 
data were organized into an analysis framework 
matrix and then summarized into the project 
tables that are included in the analysis section of 
the main report. 

The following guidelines were used to identify the 
project features that addressed or did not address 
the criteria for high-quality professional develop- 
ment. The guidelines were based on the findings 
that support the criteria (Caret, Birman, et al., 
1999, Caret, Porter, et al, 2001; Porter et al, 2000). 

For duration the documentation was analyzed for 
evidence that the project provided at least 80 hours 
of professional development. Activity type was 
determined by evidence that the project provided 



primarily reform-type activities. Such activities 
are school- or classroom-based activities that are 
incorporated into teachers’ work. 

The evidence of collective participation was identi- 
fied as opportunities for teachers from the same 
school or district to work together. Projects that 
provided opportunities for collaboration among 
participants were not identified as meeting the 
criteria, although such opportunities are noted in 
the descriptions. While such opportunities may be 
valuable, they do not address the same issues that 
are reflected in the criterion for collective partici- 
pation — fostering schoolwide or districtwide sup- 
port for teacher growth and improved instruction. 

The documentation of content focus included 
evidence that professional development was tar- 
geted on specific content knowledge, how to teach 
specific content, or how students learn. Projects 
that did not meet the criteria indicated coverage of 
multiple topic areas or failed to provide a detailed 
account of the content. 

The evidence of active learning included descrip- 
tions of opportunities for participants to observe 
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instruction or to be observed, to plan for imple- 
mentation, to analyze student work, or to present, 
lead discussions, or write reports. In particular, 
the descriptions identify projects that provided 
multiple forms of active learning, as well as proj- 
ects in which only one form of active learning was 
available. 

Coherence was assessed according to evidence that 
the projects were connected to participants’ pro- 
fessional goals or other professional development 
opportunities, aligned with standards, or provided 
opportunities for ongoing communication. A 
report that a needs assessment was conducted — a 
requirement of the program— was not consid- 
ered sufficient evidence. The documentation was 
analyzed for information about how the results 
of the needs assessment were used to design the 
project. Similarly, statements that the projects 
were aligned with standards were not considered 
sufficient evidence unless more detailed informa- 
tion was provided. 

The categories related to partnership were treated 
somewhat differently than those directly related to 
professional development. They are intended to be 
descriptive rather than indicators of quality. Two 
types of evidence of preconditions were identified: 
information about prior collaboration among the 
member organizations and at least one project goal 
that was mutually beneficial to all partners. 

There are several kinds of evidence related to 
complexity. The structure of the partnership was 
defined as either vertical or horizontal — it must be 
noted that one type of structure is not considered 
more effective than the other. Sector complexity 
was determined by identifying member organiza- 
tions from fields outside of education. Spatial com- 
plexity is a feature of projects that attempted to 
serve participants from a large geographic region. 

The category of interdependence addresses how 
the partners organized their work. It is possible 
for projects to be placed in more than one category 
because the partners may use reciprocal inter- 
dependence for some aspects of their work and 



sequential or pooled interdependence for others. 
Like complexity, one type of interdependence is 
not considered more desirable than the others. 
Evidence related to communication was identified 
as information about how frequently the partners 
interacted with each other. 



Analysis framework — evaluation 

The analysis of the evaluations of Title IIB MSP 
projects used a matrix of evaluation criteria devel- 
oped by SRI International in the Online Evaluation 
Resource Library based on the program evalua- 
tion standards established by the Joint Committee 
on Standards for Educational Evaluation (JCSEE, 
1994). The program evaluation standards focus 
on the utility, feasibility, propriety, and accuracy 
of evaluations of educational programs. Utility 
standards address the extent to which an evalua- 
tion is designed to provide information that serves 
the needs of the intended users of the evaluation. 
Eeasibility standards address the extent to which 
an evaluation is practical, includes the perspec- 
tives of all stakeholders, and is cost effective. 
Propriety standards focus on the legal and ethical 
implementation of an evaluation to ensure that 
the welfare of stakeholders is sufficiently regarded. 
Accuracy standards address the technical quality 
of the evaluation to ensure that the processes used 
and information sources are sufficiently rigorous 
and sensitive to validly represent the program 
being evaluated. 

The Online Evaluation Resource Library web site 
includes three matrixes focused around the qual- 
ity criteria of evaluation plans, instruments (and 
processes), and reports that provide a descrip- 
tive framework of what should be included in an 
evaluation that meets the program evaluation 
standards. Eor instance, one evaluation design 
component involves the methodological approach 
used in an evaluation. The Online Evaluation 
Resource Library tools include a definition of 
the methodological components, quality criteria 
related to appropriate methodological approaches, 
and an articulation of the connection to the pro- 
gram evaluation standards. 
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TABLE D4 

Analysis framework^ — evaluation 


Key feature and definition 


Subcriteria 1 


Project description 

Inclusion in the evaluation of sufficient 
information about the project and key issues 
addressing implementation. 


• Project features (goals and objectives tied to activities). 

• Stakeholders (identification of key people in the project). 

• Project context (factors that might have influenced project 
implementation). 


Evaluation overview 

Extent to which the evaluation provides 
sufficient information and alignment with goals, 
objectives, and activities. 


• Evaluation purpose (analysis of goals of project for formative and 
summative uses). 

• Evaluation questions (specific questions that align the evaluation with 
goals and stakeholders' use of information. 

• Evaluator credibility (identification and qualifications of evaluator to 
establish trust). 

• Stakeholder involvement (information about how stakeholders were 
involved in the evaluation plan and implementation). 


Evaluation design 

Information about how the evaluation was 
designed and implemented. 


• Methodological approach (how the evaluation links goals, evaluation 
questions, and data sources to present a coherent analysis). 

• Information sources and sampling (identification of groups and 
individuals that will contribute data to the evaluation). 

• Instruments (nature of the tools used to gather information). 

• Data collection procedures and schedule (indication of when, where, 
and how data collection was structured). 


Data analysis 


• Quantitative analysis (methods used to analyze quantitative data to 
answer evaluation questions). 

• Qualitative analysis (methods used to analyze qualitative data to answer 
evaluation questions). 



Source: Adapted from the Online Evaluation Resource Library, Edys Quellmalz, Stanford Research Institute, Menlo Park, CA, http://oerLsri.com/home.html. 



The analysis of the Title IIB MSP evaluations 
focused most directly on the quality criteria of 
evaluation plans as the documentation included 
in the funded proposals and first-year evaluation 
reports did not provide sufficient detail to examine 
the instrument quality criteria. The criteria for 
evaluation reports are essentially identical to the 
criteria for evaluation plans, though the connec- 
tions to the program evaluation standards are 
somewhat different (table D4). 

Data analysis 

Proposals and year one evaluation reports were 
examined for evidence that the above compo- 
nents were addressed. For instance, the proposal 
was examined to determine whether the evalua- 
tion process was structured to provide informa- 
tion that would be used formatively to improve 
the project. The year one evaluation report was 
examined to determine the extent to which the 
project staff indicated that they used the results 



of the evaluation to make project modifications. 
Matrixes were designed to ascertain the evidence 
of the components and the descriptive tables were 
built to summarize that evidence. These descrip- 
tive tables supported the summaries of quality 
of the evaluation plans and implementations as 
described in the next section. 



Summarizing the qualitative analyses 

At the request of the Institute of Education Sci- 
ences and the Northwest Regional Educational 
Laboratory external Technical Working Group, the 
authors developed a summary of the qualitative 
analysis in order to look at patterns in profes- 
sional development and evaluation designs across 
the Title IIB MSP projects in the five-state region. 
To create this summary, the authors iteratively 
assigned projects to categories based on the avail- 
able information related to the dimensions of 
the analysis framework. This categorization was 
determined by the extent to which the project 
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addressed the particular professional development 
or evaluation design issue. 

The categories that were used to summarize the 
qualitative analysis included the extent to which the 
criteria were addressed, or reported. For example, a 
component of a project was assigned to the “Ad- 
dressed” category if information related to the 
dimension was explicitly put forth in both the pro- 
posal and the annual report and evidence was found 
to support that aspect of professional development 
or evaluation. The “Somewhat addressed” category 
was assigned if the criterion was treated minimally 
or the information provided was incomplete. For ex- 
ample, in many of the proposals and annual reports, 
the evaluator was identified, but no information was 
provided about that person s experience to establish 
credibility. The category “Not addressed” indicates 
that there was evidence that the project did not 
include the aspect of professional development or 
evaluation, while “Not reported” indicates that the 
documentation does not provide sufficient informa- 
tion to assign the project to a category. More detail 
about how categories were assigned is provided in 
the summary sections for professional development 
and evaluation in the main report. 



Limitations to data sources and analysis procedures 

This analysis was based on the available docu- 
ments produced by the states and funded projects 
and an interview structured around core pro- 
grammatic issues of the professional development 
and evaluation designs. The documents included 
state requests for proposals, funded proposals, and 
year one annual reports. These are static docu- 
ments that did not represent the projects as they 
unfolded and that indicate the starting conditions 
of the projects. The interview data were collected 
before the current analysis began and are also 
limited in the extent to which they are representa- 
tive of the projects as they unfolded. 

The qualitative data analysis procedures included 
only the two authors who independently coded 
the documents and interviews and then iteratively 
identified key lines of evidence of each of the cri- 
teria. The authors have considerable experience in 
the design, implementation, and evaluation of pro- 
fessional development for science and mathemat- 
ics teachers. However, the authors recognize that 
an expert panel analysis would have been stronger 
methodology to conduct this research. 
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APPENDIX E 

COMPARISON OF PROFESSIONAL 
DEVELOPMENT CRITERIA: COUNCIL OF CHIEF 
STATE SCHOOL OFFICERS AND NORTHWEST 
REGIONAL EDUCATIONAL LABORATORY 



1 Key feature 


Council of Chief State School Officers 


Northwest Regional Educational Laboratory I 


Program duration & frequency 


Activity type 


Not included 


• Traditional activities (within-district 
workshops or conferences; courses for 
college credit; out-of-district workshops or 
conferences) 

• Reform activities (teacher study groups; 
teacher collaboratives, networks, or 
committees; mentoring; internships; 
resource centers) 


Collective participation 
among teachers 


• Participate in professional development 
with other teachers from their school 

• Participate in professional development 
with teachers from their department or 
content area 

• Participate in professional development 
with teachers from the same grade level 


• Opportunities for participants to work 
with other teachers from the same school 
or district 


Content focus 


• Provides teachers with study of subject 
content or concepts of math (or science) 

• Provides pedagogical content knowledge 
in math (or science) — how to teach 
content to students in classroom and how 
to teach cooperative learning 

• Addresses identified content weaknesses 
or needs of teachers 


• Emphasis on content knowledge 

• Emphasis on how students learn specific 
content 

• Emphasis on methods of teaching specific 
content 


Active learning 


• Demonstrating, modeling, or leading 
instruction, presenting or leading 
discussion 

• Developing or writing curricula or lessons 

• Coaching or mentoring in the classroom 

• Developing assessments or reviewing and 
scoring assessments of student work 

• Has teachers observing other teachers 

• Has teachers engaged in learning network 
or in regular discussion with teachers 


• Observing or being observed 

• Planning for classroom implementation 

• Reviewing student work 

• Conducting presentations or 
demonstrations, leading discussions, 
writing reports or plans 


Coherence of professional 
development 


• Consistent with school curriculum or 
learning goals for students 

• Aligned with state or district standards for 
learning or student performance 

• Designed to meet state teacher 
certification or licensure standards 

• Consistent with state rules for highly 


• Connected to other professional 
development and teachers' goals 

• Aligned with standards and assessments 

• Supports teachers in developing 
sustained, ongoing communication 



qualified teachers under the No Child Left 
Behind Act 
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